#DROPSHARE APP UPDATE#
#DROPSHARE APP FREE#

This feature proved well worth the effort while implementing automatic expiry, as expired files are instantly reflected in the UI.

We currently have a worker running every night to make absolutely sure each and every file is found in our database, it yet has to find a difference. Switching from our cached list to this allowed file uploads to show almost immediately while reducing the complexity of our logic immensely.

#DROPSHARE APP UPDATE#

We then update our internal files database with all information that is needed for displaying the file list.

This function immediately messages us with the details of the change. A simple lambda function is called on each add/delete to S3. To solve this once and for all we resorted to AWS Lambda processing S3 triggers. Adding caching helped the situation for a while but meant we ran into all sorts of issues (users deleting a file or changing something meant our web UI was way off most of the time). This proved unbearably slow during the first extended tests with real amounts of uploads. The first naive implementation of this feature was just doing the S3 list call for the users prefix as the file browser was opened. We decided we need to show a list of all files that have been uploaded for a user. Those stats are interpreted by our main Ruby on Rails app in workers that check the usage stats regularly, disabling file delivery if limits are overstepped. This information is put in a few different sorted sets in Redis for tracking the traffic of every file of a a user, also broken down per month. A worker is constantly following the Varnish NCSA log, parsing it for the bytes transferred in the client body. We retrieve storage stats from S3, traffic stats are generated on our side.

Our plans are limited on storage and transfer. To be fully encrypted we take advantage of Stunnel, which we use to build a transparent TLS tunnel to S3, which Varnish itself doesn’t support. Varnish also renders custom error pages on S3 and has a little VCL logic to determine acceptable caching times. Once we started serving files our own it was a wise choice to implement some form of caching. OpenResty offers an ingenious way of loading SSL certificates dynamically during the SNI handshake which we use to fetch them from a local Redis readslave. To offer this service we leverage LetsEncrypt and dynamically request certificates for custom domains.

#DROPSHARE APP FREE#

SSL is included in all our plans, the “Power” level also offers free SSL certificates on a users custom domain. Requests for our shared domain are more or less passed to varnish and S3 directly, for custom domains we resolve the domain to the users prefix on S3 and then hand off to varnish. We use OpenResty for flexible proxying as well as dynamic serving of SSL certificates. They are based on a custom stack built around OpenResty and Varnish, as well as custom statistics collection software. To proxy the files we operate caching proxies. We need to control the URL for the looks - We want to offer SSL - We want to offer custom domains (with free SSL certificates) - We need quick and granular billing Storing files is done, next part is delivering them for people to see. The credentials have a limited time of validity and are restricted to only operate on the users prefix on S3. We operate a webservice that allows the app to request access key and secret for a users path on S3 via HTTPS. It’s configurable via the normal AWS IAM policies, so it’s possible to create credentials that are only allowed to access parts of bucket by restricting the access rights to a prefix of the bucket. The service allows creating temporary, limited-privilege login credentials for AWS services. Luckily there is a workable solution: Creating temporary credentials via AWS Security Token Service (STS). This quickly turned out to be a wrong assumption as creating more than 100 buckets is not easily possible. We started out under the assumption that we could create a bucket per user and have S3 deliver the files. It’s highly reliable, reasonable priced and equipped with awesome APIs. AWS S3 was the obvious option for storing files.

Dropshare app