Mauri de Souza Meneguzzo
69933cbe35
aws: reuse session and S3 manager
...
The AWS session and s3 manager are concurrent safe, it should be reused
whenever possible:
Sessions are safe to use concurrently as long as the Session is not
being modified. Sessions should be cached when possible, because
creating a new Session will load all configuration values from the
environment, and config files each time the Session is created.
See https://pkg.go.dev/github.com/aws/aws-sdk-go/aws/session
Currently, an aws session and s3 client/manager are created every time a
call to Upload, CurrentID or Download is made. I changed it so it creates
one session and S3 manager during app startup and reuse it afterwards.
8 months ago
Philip O'Toole
cfefc090b0
Remove use of deprecated ioutil
8 months ago
Philip O'Toole
76dfa7cdea
Remove unneeded files
8 months ago
Philip O'Toole
078d0eacaf
Better timeout handling
8 months ago
Philip O'Toole
f324e6b404
Move from sums to strings
8 months ago
Philip O'Toole
a005620db4
Refactor and enhance upload logging
8 months ago
Philip O'Toole
aa2cf2dd46
Configure Store Provider
8 months ago
Philip O'Toole
f9ac68387c
More consistent use of uploader sums
8 months ago
Philip O'Toole
7472f6a949
Fix up tests
8 months ago
Philip O'Toole
5f1d98a0e2
Use Canonical Header keys for AWS metadata
8 months ago
Philip O'Toole
08080883a9
Various fixes -- test still failing though
8 months ago
Philip O'Toole
7c11295863
End-to-end test for skipped uploads due to sum
8 months ago
Philip O'Toole
22c42eed0a
Bit more uploader unit test
8 months ago
Philip O'Toole
24bfc82230
Unit test checksum compare
8 months ago
Philip O'Toole
c8e29c2269
Simpler cloud sum check logic
8 months ago
Philip O'Toole
48a000dc92
Test Storage client defaults to "no sum error"
8 months ago
Philip O'Toole
f3c1109325
More comments
8 months ago
Philip O'Toole
0d81e6f438
Fetch sums from S3 and compare
8 months ago
Philip O'Toole
58fbe3ee81
Initial framework for setting sums
8 months ago
Philip O'Toole
602d2874ca
Move to times
9 months ago
Philip O'Toole
b111a7bf98
Provide should also return last modified
9 months ago
Philip O'Toole
14858a37d4
Use last DB mod time in Provider
9 months ago
Philip O'Toole
2db982f8e8
Put sum check back in
...
This is best, until Provider is perfect.
9 months ago
Philip O'Toole
7dcfea235c
Move to move efficient index-checking for upload
9 months ago
Jason Tackaberry
7e0c25c553
Log when auto backups are completed
9 months ago
Jason Tackaberry
e60a0b54fc
Allow configurable S3 path style for auto backups
...
AWS has deprecated the path-style URLs but it's required for some
S3-like alternatives such as Minio.
9 months ago
Philip O'Toole
9e522c948d
Test uploads a little slower
...
Clearly the ticker fired 3 times before the test finished. So bump the
interval to a second, that should be enough time for the test to finish.
See https://circleci.com/api/v1.1/project/github/rqlite/rqlite/32167/output/105/0?file=true&allocation-id=659065e5aced6d10c47b2ff9-0-build%2FABCDEFGH
9 months ago
Philip O'Toole
882f19df00
Move some downloading code to restore package
9 months ago
Philip O'Toole
ad916270d2
Bring go mod import path into compliance
9 months ago
Philip O'Toole
e7ab1389c9
Periodically log boot progres
9 months ago
Philip O'Toole
e594b7a195
Ensure Stats are always reset for a test
9 months ago
Philip O'Toole
7e05f42fe3
Possible fix for racy uploader test
...
See https://app.circleci.com/pipelines/github/rqlite/rqlite/3648/workflows/d1235267-093d-4ac7-8d8d-061bc2ac9744/jobs/26157
9 months ago
Philip O'Toole
a0bf10b021
Move VACUUM to DB layer
10 months ago
Philip O'Toole
12f3dd913a
Even better package name
1 year ago