datastorage

func DeleteBucket ¶

func DeleteBucket(db *sql.DB, bucketID string, store sharding.ShardStore, logger *zap.Logger) error

Delete a bucket

func DeleteObject ¶

func DeleteObject(db *sql.DB, bucketID, objectID string, store sharding.ShardStore, logger *zap.Logger) error

This should delete all versions of an object

func DeleteObjectByVersion ¶

func DeleteObjectByVersion(db *sql.DB, bucketID, objectID, versionID string, store sharding.ShardStore, logger *zap.Logger) error

func DeleteVersionShardsAcrossNodes ¶

func DeleteVersionShardsAcrossNodes(
	bucketID, objectID, versionID string,
	store sharding.ShardStore,
	cfg *config.Config,
	logger *zap.Logger,
) error

func IsValidZipFile ¶

func IsValidZipFile(filePath string) (bool, error)

IsValidZipFile checks if the given file is a valid ZIP archive

func LookupStorageNodes ¶

func LookupStorageNodes(logger *zap.Logger) ([]string, error)

LookupStorageNodes uses the discovery service to find storage nodes for a given key

func NewRetrieveData ¶

func NewRetrieveData(db *sql.DB, bucketID, objectID, versionID string, store sharding.ShardStore, cfg *config.Config, logger *zap.Logger) ([]byte, string, error)

NewRetrieveData fetches an object from storage nodes and reconstructs it The function looks up metadata, retrieves shards from storage nodes, reconstructs the data using erasure coding, then decrypts and decompresses it

func NewStoreData ¶

func NewStoreData(
	db *sql.DB,
	data []byte,
	bucketID, objectID, filePath string,
	store sharding.ShardStore,
	cfg *config.Config,
	locations []string,
	logger *zap.Logger) (string, map[string]string, []string, error)

NewStoreData stores an object by distributing it across storage nodes The function compresses, encrypts, shards, and distributes data across storage nodes It records metadata about shard locations and proofs in the database

func NewStoreDataWithVersion ¶

func NewStoreDataWithVersion(db *sql.DB, data []byte, bucketID, objectID, versionID, filePath string, store sharding.ShardStore, cfg *config.Config, locations []string, logger *zap.Logger) (string, map[string]string, []string, error)

StoreDataWithVersion stores data with a specified version ID It follows the same flow as StoreData but uses the provided version ID instead of generating a new one

func RetrieveData ¶

func RetrieveData(db *sql.DB, bucketID, objectID, versionID string, store sharding.ShardStore, cfg *config.Config, logger *zap.Logger) ([]byte, string, error)

RetrieveData fetches an object from a bucket and reconstructs it RetrieveData uses erasure-coding to implement fault-tolerance for lost shards During retrieval, the shards are reconstructed As long as we have enough shards (in this case at least 4 of 6 shards) the reconstruction should be successful The reconstrcuted data is decrypted, then decompressed

func Retry ¶

func Retry(attempts int, sleep time.Duration, logger *zap.Logger, fn func() error) error

Retry executes the given function fn and retries it in case of an error. It uses exponential backoff for the retry intervals.

func StoreData ¶

func StoreData(db *sql.DB, data []byte, bucketID, objectID, filePath string, store sharding.ShardStore, cfg *config.Config, locations []string, logger *zap.Logger) (string, map[string]string, []string, error)

StoreData stores an object inside a bucket StoreData only works for a valid bucket, an invalid bucket would return an error The files to be stored are provided an objectID and a versionID The files to be treated are first compressed After compression, they are encrypted Successful encrypted data is then sharded and sent to their respective locations

func StoreDataWithVersion ¶

func StoreDataWithVersion(db *sql.DB, data []byte, bucketID, objectID, versionID, filePath string, store sharding.ShardStore, cfg *config.Config, locations []string, logger *zap.Logger) (string, map[string]string, []string, error)

StoreDataWithVersion is an alternative function to StoreData It takes a pre-defined object version instead of defining it locally This allows it cater for instances where a pre-defined object version has been provided

func Unzip ¶

func Unzip(source, target string) error

Unzip extracts the contents of a zip file to the specified target directory.

func UpdateShardLocations ¶

func UpdateShardLocations(cfg *config.Config, bucketID, objectID, versionID string, newLocations map[string]string) error

This adds property for partial override To get the new location we would rerun the lookup and updae the locations on the database

func ZipDirectory ¶

func ZipDirectory(source, target string) error

ZipDirectory compresses the specified directory into a zip file.