Amazon RDS Overview
So let’s get started with an overview of AWS RDS.
RDS stands for Relational Database Service and what it means is that it’s a managed database service for database that use SQL as a query language.
So SQL is a structured language to query database. It’s very well adapted and it’s running on many engines. So it allows you to create databases that RDS service in the cloud, and these databases will be managed by AWS and you get a lot of benefits from it.
So what types of database engine are managed by AWS?
Well, the first one is PostgreSQL. Then we have MySQL, MariaDB, Oracle, Microsoft SQL Server, and finally Aurora, and we have a dedicated section to Aurora. So right now you can just forget about it, okay.
But so the first five, PostgreSQL, MySQL, MariaDB, Oracle, and Microsoft SQL server, you have to remember them.
Advantage over using RDS versus deploying DB on EC2
So why would we use RDS versus deploying our own database service on top of, say, an EC2 instance, because this is possible.
Well, RDS is a managed service, and as such, AWS provides a lot of services on top of just giving us a database.
For example, the provisioning of the database is fully automated and so is the underlying operating system patching. There is continuous backups being made and you are able to restore to a specific timestamp. It’s called Point in Time Restore.
You can also have monitoring dashboards to view the performance of your database. You can have read replicas, and we have a dedicated lecture on replicates in this course. To improve read performance, you can set up a multi AZ, and so we do have also sections on multi AZ that will be helpful for disaster recovery.
You have maintenance windows for the upgrades and you have scaling public capabilities, both vertical, by increasing the instance type, and horizontal, by adding read replicas.
Finally, the storage is backed by EBS.
So this is something we already know, which is gp2 volumes or io1. But the only thing we do not have, is we cannot SSH into the instances, the RDS instances.
So because this is a managed service, AWS provides us a service and we don’t have access to the underlying EC2 instance. But this is not too bad because we get all these things that we would have to set up on our own if we wanted to deploy our own database engine on EC2.
Storage Auto Scaling
Let’s talk about something related to RDS storage auto scaling.
So the idea is that when you create an RDS database you need to say how much storage you want.
For example, you want 20 gigabytes of storage, but say you are using your database a lot and you’re about to run out of free space, then with this feature enabled, the RDS storage auto scaling, RDS will detect that and will scale automatically the storage for you. So you don’t have to do any kind of operations such as taking down database to increase the storage.
So the idea is that your application does a lot of read and writes to your RDS database, and then automatically, with some threshold we’ll see in a second, then the storage can auto scale. And that is a feature of RDS that is a very, very nice feature. So this all is meant to make you avoid the operation of scaling manually, your database storage.
For this, you need to set a maximum storage threshold, so maximum limits for how much you want the storage to grow, ’cause you don’t want it to grow infinitely, maybe, and you can automatically modify storage if the free storage is less than 10% of what has been allocated and the low storage has been lasting for more than five minutes and six hours have passed since the last modification.
If that’s the case then the storage will auto increase when you enable it.
This is very helpful for applications which that have a unpredictable workload and this supports all database engines for RDS, just as MariaDB, MySQL, PostgreSQL, SQL server, and Oracle.
RDS Read Replicas for read scalability
Going into the exam is going to be extremely important to understand the difference between RDS Read Replicas and Multi AZ and understand exactly the use cases for those. So this lecture is really dedicated to understanding Read Replicas and Multi AZ.
So let’s get started with the Read Replicas. Read Replicas as the name indicates help you to scale your reads.
So let’s have an example.
Here, we have our application and we have an RDS database instance and our application performs reads and writes to our database instance, but say we want to scale the reads because that’s, main database instance can not scale enough.
It receives too many requests.
Well, we can create up to 15 Read Replicas and they can be within the same availability zone, cross availability zone, or cross region. So three different options and they’re very important to remember.
So say we have another RDS instance Read Replica and another one and what will happen is that they will be an asynchronous replication between the main RDS database instance and the two Read Replicas.
Asynchronous, that means that the reads are eventually consistent.
What that means is that, for example, if your application reads from the Read Replica before they had the chance to replicate the data then you may get all data. And this is why it’s called eventually consistent asynchronous replication.
And these replicas, they can be awesome for reading for scaling reads but they can also be promoted to their own database. So you can take one of these replicas and say, okay I want it to be its own database now and take rights.
And so you promote it to its own database. It’s completely out of the replication mechanism after that but it’s lives and has its own life cycle afterwards.
So in case you wanna use Read Replicas, the main application in orange in the top of the screen must update the connection string to leverage the list of all the Read Replicas you have in your RDS cluster.
RDS Read Replicas — Use Cases
So let’s talk about a classic use case for you to Read Replica.
So in this example, we have a production database and it is taking on normal load.
Our production database is having read and writes to our main RDS database instance. And the new team comes in and say, we want to run some reporting and some analytics on top of your data.
And so if you plug in that reporting application onto the main RDS database instance then it’s going to overload it and possibly slow down the production application and you don’t want that. So instead what you do as a solutions architect is you create a Read Replica to run the new workload there. So you create a Read Replica.
There is some asynchronous, a replication that happens between your main RDS DB database instance and your Read Replica and then your reporting application can just do reads from your Read Replica and run the analytics there.
The production application is completely unaffected in this case and that’s perfect. So remember, if you have a Read Replica you need to ensure that it is only for SELECT type of statements and SELECT is a SQL keyword. And so you can not use keywords such as INSERT, UPDATE or DELETE, which change database itself.
RDS Read Replicas — Network Cost
So, let’s talk about the networking cost associated with RDS Read Replica.
So in AWS, you should know that’s normally there is a cost when the data goes from one availability zone to another but there are exceptions and these exceptions are usually for managed services. So for RDS Read Replica, this is a managed service.
If your Read Replica is within the same region, okay? Different AZ, but same region, you’re not paying that fee.
So what this means that if you have a RDS DB instance in us-east-1a and then a Read Replica in us-east-1b, and there is asynchronous replication because this is a Read Replica, even though the traffic goes from one AZ to a different AZ that replication traffic is going to be free because RDS is a managed service and they give you that traffic for free.
But if you are using a cross region replica, so you are in one region, us-east-1 and you go to another region, eu-west-1, for example then your RDS DB instance and your Read Replica will have replication that will go across regions and this will incur a replication fee for your network.
RDS Multi AZ (Disaster Recovery)
Finally, let’s talk about RDS Multi AZ and Multi AZ is mainly used for Disaster Recovery.
So we have our application and it performs the reads and writes to our Master database instance which is in availability zone A and what we thinks you have is a synchronous replication to a standby instance in AZ B, and they will replicate every single change in the Master synchronously.
So that means that when your application writes to the Master, that change needs to also be replicated to the standby to be accepted. And so what we get out of it is one DNS name.
So your application talks to one DNS name and in case there is a problem with the Master, there will be an automatic failover to the standby database. Thanks to that one DNS name.
So thanks to this, we increase the availability.
This is why it’s called Multi AZ and they will be failover in case we lose an entire AZ or we lose the network or there is an instance or storage failure for the Master database.
In which case, obviously the standby database will become the new Master.
He don’t need to do any manual intervention in your apps as long as it tries to keep on connecting to your database automatically at some point, it will fail over to the standby that will be promoted as the Master, and you’ll be good and it’s not used for scaling.
So as you can see here, the standby database is just for standby. No one can read to it. No one can write to it, it’s just here as a failover in case anything goes on with your Master database.
So, quick question is, is there a possibility to have the Read Replicas being set up as Multi AZ for Disaster Recovery? And the answer is yes, you can set up your Read Replica as a Multi AZ if you want it to.
So that’s it for the difference between Read Replicates and Multi AZ.
RDS — From Single-AZ to Multi-AZ
So a question is around how do we make an RDS database go from Single AZ to Multi AZ?
So what you should know is that it is a zero downtime operation. That means you do not need to stop the database to go from Single AZ to Multi AZ. The only thing you need to do is to click on modify for the database and enable Multi AZ.
That means that your RDS database instance is going to go to have a Master, to have a standby DB with synchronous replication without you doing anything except modifying that setting and the database will not stop.
Now, this is what the exam will ask you but I wanna show you what will happen behind the scenes for it to work.
So, the following will happen internally.
There will be a snapshot taken by RDS automatically of your main database and that this snapshot will be restored into a new standby database, okay?
And then once the standby database is restored, there will be synchronization established between the two database and so therefore the standby database will catch up to the main RDS database and there you go, you will be in a Multi AZ setup.
hands on..
RDS Custom
So now let’s do a short lecture on RDS Custom.
So, we know that with RDS we don’t have access to any underlying operating system, or customization, but with RDS Custom we actually do.
And RDS Custom is for two database types, it’s for Oracle and Microsoft SQL Server. In which, thanks to RDS Custom, we have access to the OS, and database customization. So with RDS, we still get all the benefits of automated setup, operations, and scaling of the database in AWS. But with the RDS Custom option, the custom part of RDS Custom, we actually have access to the underlying database and operating system.
So we can actually configure internal settings, install patches, enable native features, and we can actually access the underlying EC2 instance behind RDS, using SSH or SSM Session Manager.
So here is my EC2 instance, and as a user, we can now, if it’s using Amazon RDS Custom, apply customizations and SSH into it.
To perform any customization it is recommended to deactivate the automation mode, so that RDS does not perform any automation, or maintenance, or scaling, or whatever you want, while you perform your customization.
And also because you may actually break things, because now you have access to the underlying EC2 instance, it is much better to take a database snapshot, otherwise you will not be able to recover from your actions.
So to summarize, RDS versus RDS Custom.
RDS will manage your entire database, and your operating system.
Everything is managed by AWS, you don’t have to do anything, and RDS Custom, it’s for Oracle and Microsoft SQL Server only, and you have actually full admin access to the underlying operating system and the database.
Amazon Aurora
So let’s talk about Amazon Aurora.
Now, you don’t need deep knowledge on it, but you need enough high level overview to understand exactly how it works.
Aurora is going to be a proprietary technology from AWS.
It’s not open sourced, but they made it so it’s going to be compatible with Postgres and MySQL and basically your Aurora database will have compatible drivers. That means that if you connect as if you were connecting to your Postgres or MySQL database, then it will work.
Aurora is very special and I won’t go too deep into the internals, but they made it cloud optimized and by doing a lot of optimization and smart stuff, basically they get 5x performance improvements over MySQL on the RDS or 3x performance of Postgres on RDS.
Not just that, but in many different ways, they also get more performance improvements. To me, I’ve watched it, it’s really, really smart but I won’t go into the details of it.
Now Aurora storage automatically grows and I think this is one of the main features that is quite awesome. You start at 10GB, but as you put more data into your database it grows automatically up to 128TB. Again, this has to do with how to design it but the awesome thing is that now as a DB or a SysOps, you don’t need to worry about monitoring your disc.
You just know it will grow automatically with time.
Aurora can have up to 15 read replicas, and the replication process is going to be faster than MySQL.
You will see sub 10 ms replica lag typically.
Now if you do failover in Aurora it was going to be instantaneous, so it’s gonna be much faster than a failover from Multi-AZ on MySQL RDS. And because it’s cloud-native, by default you get high availability.
Now, although the cost is a little bit more than RDS, about 20% more, it is so much more efficient that at scale it makes a lot more sense for savings.
Aurora High Availability and Read Scaling
So let’s talk about the aspect that are super important which is high availability and read scaling.
So Aurora is special because it’s going to store six copies of your data anytime you write anything across 3 AZ. And so Aurora is made such as it’s available, so it only needs four copy out of six for writes.
So that means that if one AZ is down, you’re fine.
And it only needs to have three copy out of six needed for reads. So again, that means that it’s highly available for reads.
There is some kind of self healing process that happens which is quite cool, which is that if some data is corrupted or bad, then it does self healing with peer-to-peer replication in the backend and it’s quite cool.
And you don’t rely on just one volume, you rely on 100s of volumes. Again, not something for you to manage. It happens in the backend, but that means that you just reduce the risk by so much.
So if you look at it from a diagram perspective you have 3 AZ and you have a shared storage volume but it’s the logical volume and it does replication, self healing, and auto expanding, which is a lot of features.
So if you were to write some data, maybe blue data, you’ll see the six copy of it in three different AZ.
Then if you write some orange data again six copy of it in different AZ and then as you write more and more data, it’s basically going to go again six copy of it in three different AZ. The cool thing is that it goes on different volumes and it’s striped and it works really, really well.
Now we need to know about storage and that that’s it. But you don’t actually interface with the storage.
It’s just a design that Amazon made and I want to give it to you as well so you understand what Aurora takes. Now, Aurora is like multi-AZ for RDS. Basically there’s only one instance that takes writes.
So there is a master in Aurora and that’s where we’ll take writes. And then if the master doesn’t work the failover will happens in less than 30 seconds on average.
So it’s really, really quick failover. And on top of the master you can have up to 15 read replicas all serving reads. So you can have a lot of them and this is how you scale your read workload. And so any of these read replicas can become the master in case the master fails.
So it’s quite different from how RDS works but by default you only have one master. The cool thing about these read replicas is that it supports cross region replication.
So if you look at Aurora on the right hand side of diagram, this is what you should remember. One master, multiple read replicas and the storage is gonna be replicated, self-healing, auto expanding, little blocks by little blocks.
Aurora DB Cluster
So this is more around how Aurora works when you have clients.
How do you interface with all these instances?
So as we said, we have a shared storage volume which is auto expanding from 10GB to 128GB. Your master is the only thing that will write to your storage.
And because the master can change and failover, what Aurora provides you is what’s called a writer endpoint. So it’s a DNS name, a writer endpoint, and it’s always pointing to the master. So even if the master fails over, your client still talks to the writer endpoint and is automatically redirected to the right instance.
Now, as I said before, you also have a lot of read replicas. What I didn’t say is that they can have auto scaling on top of these read replicas.
So you can have one up to 15 read replicas and you can set up auto scaling such as you always have the right number of read replicas.
Now, because you have auto scaling, it can be really, really hard for your applications to keep track of where are your read replicas.
What’s the URL? How do I connect to them?
There is something called a reader endpoint. And a reader endpoint has the exact same feature as a writer endpoint. It helps with connection load balancing and it connects automatically to all the read replicas.
So anytime the client connects to the reader endpoints it will get connected to one of the read replicas and there will be load balancing done this way.
Make sure, just notice that the load balancing happens at the connection level, not the statement level. So this is how it works for Aurora.
Remember writer endpoint, reader endpoint. Remember auto scaling. remember shared storage volume that auto expands. Remember this diagram ’cause once you get it, you’ll understand how Aurora works.
Features of Aurora
Now if we go deep into the feature you get a lot of things I already told you.
Automatic failover, backup and recovery, isolation and security, industry compliance, push-button scaling by auto scaling, automated patching with zero downtime.
So it’s kind of cool that magic they do in the backend. Advanced monitoring, routine maintenance, so all these things are handled for you. And you also get this feature called backtrack which is giving you the ability to restore data at any point of time. And actually doesn’t rely on backups, it relies on something different.
But you can always say, I want to go back to yesterday at 4:00 PM and you say, “Oh no, actually I wanted to do yesterday at 5:00 PM,” and it will work as well, which is super, super neat.
So that’s it for Aurora, and I will see you in the next lecture.
Hands On..
Aurora Replicas — Auto Scaling
So let’s talk about replica auto scaling.
So let’s say we have a client and we have three Aurora’s instances right now. So one will be writing through the writer endpoint and the other two would be reading through the reader endpoint.
Okay. Now let’s say that we are having many, many requests, read requests on the reader endpoint, and so therefore the Amazon Aurora databases have an increased CPU usage.
In that case, we can set up replica auto-scaling, and what this will do is that it will add obviously Amazon Aurora replicas, and then what will happen is that automatically the reader endpoint is going to be extended to cover these new replicas and therefore these new replicas will start receiving some traffic and the reads will be happening in a more distributed fashion, hopefully bringing down the overall CPU usage.
So this is replica auto scaling.
Aurora — Custom Endpoints
The second thing is custom endpoints.
So let’s say we have the same setup, but this time we have two different kinds of replicas. As you can see, we have some db.r3.large and some db.r5.xlarge. So some read replicas are bigger than others. And the reason you would do this is that you want to define a subset of your Aurora instances as a custom endpoint. So let’s say we define a custom endpoint on these two bigger Aurora instances.
The reason we would do so is for example, that we know that these instances are more powerful and therefore they’re going to be better to run analytical queries on these specific replicas.
So when you do so, you have a custom endpoint that’s defined. And when you have a custom endpoint, generally the reader endpoint itself is not used after defining custom endpoints. So it would not disappear, but you would not use it anymore.
And what you would do in practice in like practically speaking, is set up many custom endpoints for many different kind of workloads, therefore allowing you to query only a subsets of your Aurora replicas.
Aurora Serverless
Next we have serverless.
So this is going to give you automated database instantiation and auto-scaling based on actual usage, which is great when you have infrequent, intermittent or unpredictable workloads.
And therefore you don’t need to do any capacity planning.
You’re going to pay per the second of each Aurora instance that is being spun up and this can be a lot more cost effective.
So how does that work?
Well, the client is going to talk to a proxy fleet that is managed by Aurora and in the backend, many Aurora instances will be created based on your workload in a serverless fashion, so you don’t have to provision capacity at all in advance.
Global Aurora
Finally, we have Global Aurora.
So in case we have a Aurora cross region read replica it’s going to be very helpful for disaster recovery. Very simple to put in place. But you can also set up the Aurora global database, which is the recommended way of doing things today.
In this case, you have one primary region where all the reads and writes happen, but you can set up up to five secondary read-only regions, where the replication like should be less than one second and up to 16 read replicas per secondary region.
And this will help you decrease latency for the read workloads all over the world, but also in case you have a database outage in one region, promoting another region for disaster recovery purposes has an RTO to recover a time objective of less than one minute. So it takes us in one minute to recover into another region.
And a sentence you should look out for from an exam perspective, it takes on average less than one second to replicate data across region for your Aurora global database.
So let’s have a look. We have us-east1 as our primary region, which is where the applications do the read and writes, and then we’ll set up a secondary region in us-one or some replication happening with a global database of Aurora and the applications that it can do read only from that setup.
But in case us-east1 fails, then we can fail over to us-one by promoting it as a read write Aurora cluster.
Aurora Machine Learning
Aurora also has an integration with machine learning services within AWS.
So the idea with Aurora Machine Learning is that you can have ML based predictions to your applications via the SQL interface. It’s a simple, optimized and secure integration between Aurora and different AWS machine learning services.
So two supported services are SageMaker, which allows you to use any kind of machine learning model in the backend or Amazon Comprehend if you want to do sentiment analysis.
Now, you don’t need to be an expert in SageMaker or Comprehend, you just need to know that Aurora has an integration with those. So in order to use Aurora Machine Learning, you don’t have to, you don’t need to have a machine learning experience.
And the use cases for this would be to have fraud detection, ads targeting, sentiment analysis or product recommendation all within Aurora. So to give you the architecture idea, Aurora is going to be connected to the machine learning services in AWS. And your application can just run a very simple SQL query.
For example, what are the recommended products?
Aurora will send some data into the machine learning services such as, for example, the user’s profile, the shopping history, et cetera, et cetera. And then the machine learning service will return a prediction directly to Aurora.
For example, the user should buy a red shirt and blue pants, and then Aurora can just return the query results to the application all through as a result of the SQL query, which is very handy.
RDS Backups
So now let’s talk about RDS Backup.
So first of all, you have automated backups and what it means that this service, RDS service, will automatically do a daily full backup of the database during your database backup window.
But on top of it, every five minutes there’s going to be the transactions log being backed-up. That means that the earliest backup you have is five minutes before now.
That means that from these automated backups you can restore to any point of time up until five minutes ago. And for the automated backup retention, you can say between one to 35 days of retention. And if you wanted to disable this feature you would set zero to disable automated backups.
The second option is to do manual DB Snapshots, manual database snapshots. And these, as the name indicates, are manually triggered by the user. But the benefit of it is that this backup, that you do manually, you can retain it for as long as you want. So automated backups expire, whereas DB Snapshots that are manual can be retained for as long as you want.
So here is a trick.
When would you use backups, obviously, to back up your data, but also if you want to save some costs. So say for example, you have an RDS database and you know that you’re going to use it only two hours per month.
Instead of stopping the database in which, for example, you will still have to pay for the storage even though your database is stopped. What you can do is that after using it for two hours you do a snapshot and then you delete the original database.
The snapshot is costing you way less than the actual storage of the RDS database. And when you’re ready to use again the database you would restore the snapshot and use that.
Aurora Backups
Now for Aurora backups, it is actually quite similar.
You have automated backups from one to 35 days but they cannot be disabled. So on RDS, they could be disabled, but not on Aurora. And you have a point-in-time recovery at any point of time in that timeframe, so you can recover to whatever you want.
Now for manual DB Snapshots, they can also be manually triggered by the user and you retain them for as long as you want. So you can see Aurora and RDS Backups are quite similar.
RDS & Aurora Restore options
Now what about the restore options?
The first thing is that you can restore an RDS or Aurora backup or snapshots, they’re the same thing, into a new database. So anytime you restore an automated backup or a manual snapshot, then it will create a new database.
The other option is that you can restore a MySQL RDS database from S3. And S3, we haven’t seen it yet, it’s a way to store objects in the cloud on AWS, it’s a very popular service, And the idea is that you will create a backup of your on-premises database, and then you will place it onto Amazon S3, which is some object storage.
And finally, there is the option in RDS to restore the backup file from Amazon S3 onto a new RDS instance running MySQL.
And if you wanted to, instead, restore to MySQL Aurora cluster, what you can do is that you take a backup again of your on-premises database, so externally, and then you used to use a software called Percona XtraBackup.
It only works with this for now.
Then the backup file of Percona XtraBackup goes onto Amazon S3. And from this you can restore the backup file onto a new Aurora cluster running MySQL.
So the difference here is that for restoring into RDS MySQL you just need a backup of your database for restoring into Aurora MySQL. What you need to do is to do a backup using Percona Xtrabackup, and then from S3 into your Aurora DB cluster.
Okay, so those are restore options for either the snapshots or the database dumps onto Amazon S3.
Aurora Database Cloning
And finally, there’s a last feature you need to know about called Aurora Database Cloning. So this allows you to create a new Aurora database cluster from an existing one. So say you have a production database in Aurora and you want to actually run test on it but you want to copy the data because your test application must be in a staging environment.
Well, for this you’re just going to clone your production Aurora database. It’s going to give you a new database, and you can call it, for example, the Staging Aurora database.
And it’s going to be very quick. It’s faster than doing a snapshot and restore. So why, while cloning uses the copy-on-write protocol, how does that work?
Well, initially, when you create a database clone it’s going to be using the same data volume as the original database cluster, which is going to be fast and efficient because you don’t have any copying of data.
And then as the updates are being made to the production Aurora database, or to the Staging Aurora database, then new additional storage is going to be allocated and data is going to be copied and separated. So that means you have the best of both worlds.
So doing a database cloning is very fast and cost effective, and it’s extremely useful to have a staging database from a production database without impacting the production database and without going through snapshot and restore features.
We’ve seen all the options for backup and restore as well as cloning.
RDS & Aurora Security
Now, here is a quick lecture around RDS and Aurora Security.
So you can encrypt the data at-rest on your RDS and Aurora database. That means that the data is encrypted on the volumes. For this, you will be having the master and any replica encrypted using KMS. And this is defined at launch time during the first launch of your database.
If somehow you haven’t encrypted the master database, the main database, then the read replicas cannot be encrypted.
Also, if you wanted to encrypt an already existing unencrypted database, what you would have to do is to take a database snapshot from that un-encrypted database, and then you restore that database snapshot as an encrypted database. Okay?
So you have to go through a snapshot and restore operation. So this is for at-rest encryption. Then you have in-flight encryption.
So between your clients and your database. So each database on RDS and Aurora is ready to have in-flight encryption by default. And so therefore, your clients must use the TLS root certificates from AWS. They’re provided on the AWS website.
In terms of database authentication. Because this is RDS and Aurora, you can use the classic combo of username and password. But because it says AWS, you can also use IAM roles to connect to your database.
That means that’s, for example, if you’re EC2 instances had IAM roles, they can authenticate to your database directly using that and not a username and a password, which can help you manage all the security within AWS and IAM.
You can also control network access to your database using security groups. So you can allow or block specific ports, specific IP, specific security groups.
And then finally RDS and Aurora do not have SSH access, of course, because they’re managed services, except if you use the RDS custom service from AWS.
And if you wanted Audit Logs. So to know what queries are being made on RDS and Aurora over time and what’s happening on databases, you can enable Audit Logs. And then they will be lost after a bit of time.
Therefore, if you wanted to keep them for a long period of time, what you need to do is to send them into a dedicated service called the CloudWatch Logs service on AWS.
Amazon RDS Proxy
So now, let’s talk Amazon RDS Proxy.
So we know we can deploy our RDS database within our VPC, but now we can also deploy a fully managed database proxy for RDS.
So you may say, “Okay, we can access our RDS database directly.
Why do we need a proxy to access our database?” Well, if you use an As an RDS Proxy, this will allow you, your application, to pool and share the database connections established with the database.
So instead of having every single application connect to your RDS database instance, they will be instead connecting to the proxy, and the proxy will pool these connections together into less connections to the RDS database instance.
So why may you do this?
Well, this makes a lot of sense if you have a lot of connections into your RDS database instance, and it’s going to improve your database efficiency by reducing the stress on database resources, for example, CPU and RAM, and also minimize the open connections and the timeouts into your database.
So that’s one thing you need to look out for from the exam perspective. Now, as I said, the RDS Proxy is fully serverless. It’s auto scaling, so you don’t need to manage its capacity. And it’s highly available. It’s across multiple AZ.
And therefore, in case, for example, there is a failover that happens on your RDS database instance. For example, it goes from the primary instance to the standby instance, then it will, thanks to the RDS Proxy, reduce the failover time by up to 66%.
So this is true for RDS and Aurora.
So again, instead of having all your applications connect to the main RDS database instance and then handle the feller themselves, they will just connect to the RDS Proxy, which does not know any failover. And the RDS Proxy will, itself, handle the failover of the RDS database instance, therefore improving your failover time.
So the RDS Proxy supports MySQL, PostgreSQL, MariaDB, Microsoft SQL Server, as well as Aurora for MySQL and PostgreSQL. It doesn’t require you to do any code change in your application.
Just instead of connecting to your RDS database instance or your Aurora database, you’d now just connect to your RDS Proxy and you are done.
On top of it, there’s a third advantage to using an RDS Proxy. It is to enforce IAM authentication for your database. So to make sure that people can only connect to your RDS database instance using IAM.
And then these credentials can be securely stored in another service called the AWS Secrets Manager. So again, if you need to see a way to enforce IAM authentication for your database, think RDS Proxy.
And finally, the RDS Proxy is never publicly accessible. So it’s only accessible from within your VPC. So you cannot connect over the internet to your RDS Proxy, which is enhanced security.
And so one service we haven’t seen that is going for sure to be very helped by the RDS Proxy is called the Lambda functions. And the Lambda functions, they will execute pieces of code.
Okay, you don’t need to know them right now, but we have to tackle them right now because of the RDS Proxy feature. And so the Lambda functions, they can multiply, multiply many, many times. And they will appear and disappear very, very fast.
And therefore, imagine you have 100 or 1,000 Lambda functions appearing and disappearing just like that, and opening up connections into your RDS database instance, they will be a big problem because it will leave open connections and timeouts, and it will be a mess.
And therefore what you do is that you use the RDS Proxy to pool connections for your Lambda functions, and then the Lambda functions will overload the RDS Proxy, but it’s meant to be overload.
And the RDS Proxy will pool these connections into less connections to the RDS database instance, thus, solving your problems, okay?
But again, RDS Proxy is used to minimize and pool connections on your RDS database instance. It’s also used to minimize failover time and reduce it by up to 66%.
And also, it’s used to enforce IAM authentication for your database and to securely store its credentials in the Secrets Manager service.
Amazon ElastiCache Overview
Let’s talk about Amazon ElastiCache.
So the same way you get RDS to have managed relational databases, ElastiCache is going to help you get managed Redis or Memcached which are cache technologies.
So what are caches?
Well, they are in-memory databases with really high performance and low latency. And what the cache will help you with is to help reduce the load off of databases for read intensive workloads.
The idea is that the common queries are going to be cache, and therefore your database will not be queried every time. Just your cache can be used to retrieve the results of these queries.
What this also helps you do is makes your application stateless by putting the state of your application into Amazon ElastiCache. And because we have the same benefits our RDS, AWS will take the same maintenance of the operating system, the patching, the optimization the setup, configuration, monitoring, failure recovery and backups.
Now, if you use Amazon ElastiCache just so you know that it will require you to do some heavy application code changes for your application.
This is not something just enable and off you go, you have a cache. You need to change your application to query the cache before or after querying the database. And we’ll see the strategies in a minute.
ElastiCache Solution Architecture — DB Cache
So now let’s talk about the architecture for using ElastiCache.
And there can be many, but I’m just giving you an example. So, let’s say we have Amazon ElastiCache and RDS database and your application. The application will query ElastiCache to see if the queries are already being made.
And if it has already been made, and it is stored in ElastiCache, then it’s called a cache hit. And then it just gets the answer straight from Amazon ElastiCache. And we were saving a trip to the database to do the query. Now, in case of a cache miss, then we need to fetch the data from the database.
So we’re going to read it from the database.
And then for other application or other instances where the same query will be made we can write the data back into the cache so that such as the same query next time will result in a cache hit.
The idea is that it will help relieve load from your RDS database. And because you’re storing data in the cache there must be a cache invalidation strategy to make sure that only the most current data is used in there.
And this is the whole difficulty around using caching technologies.
ElastiCache Solution Architecture — User Session Store
Another architecture is around storing the user session to make your application stateless.
So the idea is that your user will do a login into any kind of your applications, and then the application will write the session data into Amazon ElastiCache.
Now, if your user is redirected to another instance of your application then your application can retrieve the session cache, the session directly from the Amazon ElastiCache And therefore your user is still logged in, and doesn’t need to log in one more time.
And so the idea is that now you made your application stateless by writing the session data of your user into Amazon ElastiCache.
ElastiCache — Redis vs Memcached
So if we compare Redis and Memcached, what do we have?
Well, for Redis, we have Multi AZ with Auto-Failover, and we have Read Replicas if you wanted to scale the reads and have high availability. So this looks a lot like RDS. We have data durability using AOF persistence, and we have backup and restore features.
So again, Redis is all about data durability. And feature wise as a cache, it supports sets and sorted sets. So here I’d like to show Redis as a cache that gets replicated that is highly available and durable.
Whereas Memcached is using multi-node for partitioning of data. So it’s called sharding. There is no high availability, there’s no replication happening. It’s not a persistent cache. There is no backup and restore. And this is a multi-threaded architecture.
So we have multiple instances altogether working in Memcached with some sharding. So the idea here that you need to remember is that Redis really is here for high availability, backup, read replicas, this kind of stuff.
Whereas Memcached is a pure cache that’s distributed where you can afford to lose the- You lose your data, there’s no high availability, and there’s no backup and restores.
So these are the main differences that you’re going to have between the two technology.
Hands On..
ElastiCache — Cache Security
Now, let’s talk about ElastiCache Security.
So ElastiCache supports IAM authentication for Redis only and for the rest you just use a username and password. If you define IAM policies on ElastiCache, they’re only used for AWS API-level security.
Then you have security within Redis called Redis AUTH, where you can set a password and a token when you create a Redis cluster, which gives you an extra level of security for your cache on top of your security groups. It also supports SSL in flight encryption.
Next we have Memcached.
So Memcached supports SASL-based authentication which is quite an advanced mechanism.
Just remember, just a name.
So to summarize, when we have an EC2 instance in a client, for example, we could use Redis AUTH to connect to Redis Cluster that’s protected by Redis Security Group and we can have in-flight encryption. Or alternatively, we could, for example, leverage IAM authentication for Redis.
Patterns for ElastiCache
Now, there are three kinds of patterns for loading data into ElastiCache.
The number one is Lazy Loading, where all the read data is cached and data can become stale in the cache, Write Through, which is to add or update data in the cache whenever it is written to a database where there’s no stale data, and using ElastiCache as a session store that we saw before and then we can expire a session using the time to live features.
Now, caching is very hard and there’s this very famous quote in computer science that “There are only two hard things in computer science: caching invalidation and naming things.”
So caching is a very complicated topic and give you a very high-level overview, but let’s illustrate the lazy loading strategy here.
So your application, if it has a cache hit in ElastiCache data from the cache, otherwise there’s a cache miss, therefore, you read from the database and you write to the cache.
This is something we’ve seen from before and it is called lazy loading because only when you don’t have a cache hit, then you load the data into Amazon ElastiCache.
ElastiCache — Redis Use Case
So now let’s talk about the use cases for Redis and there’s one you need to know that. It is around creating a gaming leaderboard.
So it is very complex.
The idea is that you wanna say who is number one, who is number two, who’s number three at any moment in time for your game. And so there is a feature in Redis called Sorted Sets which guarantees both uniqueness and element ordering.
And each time an element is added it’s going to be ranked in real time and then added in the correct order.
The idea is that if you have a Redis cluster, you can create a realtime leaderboard with a number one, number two, and number three player in realtime and all the Redis cache will have at the same leaderboard available.
That means that your clients, when talking to your Amazon ElastiCache using Redis, they can have access to this realtime leaderboard and we don’t need to program that feature application side. We can just leverage Redis with the sorted sets to get access to your realtime leaderboard.
