Saving advanced cache file across several instances #22
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
This is a somewhat specialized problem, but I thought I would spell it out and suggest a solution if you are interested in me working on this.
We have several EC2 instances running WordPress behind a load balancer. This usually works great because WP options are typically saved a the single database. In the case of WP-FFPC, though, if we make any changes to the
advanced-cache.php
, these are not saved the DB nor are they propagated throughout the EC2 instances--the changes will only save on the single instance we happened to connect with. This caused a performance issue today when we traced back sporadic site slowness to a single instance that did not pick up our changes toadvanced-cache.php
.My thought was that in addition to saving these settings to
advanced-cache.php
, WP-FFPC could save these to the database. A cron event could periodically check the value to make sure the currentadvanced-cache.php
file is up-to-date. I don't think this should be part of the plugin, because I can't imagine many people having this problem. So, it seems to me an easy and non-obtrusive way for me to do this is to just add a WP action insideWP_FFPC:: deploy_advanced_cache()
. I could hook into this outside the plugin, passing through $this->global_config to do what I need to do. Thoughts?interesting issue, good points. Adding an action to any part of the plugin should not be a problem, although the
deploy_advanced_cache
function is not ready to become static function; it uses way too much of the class' variables.The integrity check is a very good idea. It would certainly be easier and faster if the whole advanced-cache array was stored in db.
As always, I welcome any additions or commits, but I'll take a look into it as well ( when I have time, I'm a bit short on that these days :/ )
We are also running multi-instance WP and on some days it seems that WP makes an extra effort to make it as hard as possible :) I definitely like the idea of easier way to recreate cache settings from DB. On other hand, i would like BETTER if it didn't write anything to DB but just worked off the local file. Let me explain why:
WP is NOT built to be run on multiple instances. You have to maintain same php codebase (including wp-config.php) on all machines. You have to make sure your uploads/ is either on CDN or synced between nodes. You have to take into account the wp-cron either launches on random node or, if you have set it up on one node, the db behind wp-cron has no idea about running the same job on other instances. Updating WP plugins, codebase happens on one node only etc etc.
What I am trying to say here, is that for most of the plugins, which can run php and can afford DB connection, it is possible to have central management and DB syncing, but there are lot of areas in WP where central DB access just doesn't suffice.
We have experimented with many approaches. Running lsync/rsync daemons on nodes, setting up NFS, using CDN as code repository, using load balancer to drive wp-admin/upload etc to single instance, setting system cron to do cleanup/sync.. all of this is very heavily dependent on your specific network setup and still causes occasional problems.
So, we gave up. Seriously :) What we do is that we have single-instance WP set up as a staging environment, where we run all the WP/Plugin updates, where we do conf changes and testing. Once we are happy, we are packaging that wp code into application version to be deployed to all production nodes -- changing some wp-config, logging settings. There are some scripts/jenkins jobs involved, but it is pretty straightforward process. Production nodes webserver doesn't even have rights to write to any php file/directory on webroot, which makes sure everything is in sync between nodes and we have good understanding when and what was changed without need to log into each node to make sure.
The additional step tho is to log in to wp-admin and make the same change i did for ffpc setting in advanced-cache.php in the GUI to save it to db -- which i would like to get rid of..
This "problem" exists across most WP plugins. The approach @aivar gave is what we do as well. You essentially put your updated configs on a node that can sync the files out to the other nodes, and you'll be set. We use Jenkins for this, but you can have a standby node in staging or something that is able to update production once certified. There are many ways to go about it.
The only possible solution I could think about it to write a manual query into the advanced-cache.php ( which should exist, because this is what WordPress is using for serving cache with ) but that might cause issues later, with WP version changes, but it would indeed make things a bit cleaner.
I'll test this solution to see the negative impact/sideeffects.
Writing any DB-dependancy into advanced-cache is a bad idea - the whole point of cache is to make it as fast as possible. Connecting to DB.. is slow. Unfortunately I don't think there is any better solution than making sure you're propagating single configuration file to multiple nodes. I second @aivar in removing config DB-dependency altogether.
I do agree with the speed issues. Skipping this idea then.
@petermolnar
I noticed that in wp-ffpc-class.php, the config options are stored in the database:
update_site_option( $this->global_option , $this->global_config );
Could you elaborate on how that is used? I work on a relatively busy WordPress site that runs on about a dozen hosts, and noticed that changes to advanced-cache.php are not made live (I updated that file in git and pushed it to all the hosts). I had to update the settings in the UI before they were made live. Whichever host I clicked save on had its config updated. In short, the database appears to be the SOR.
I am migrating from W3TC, which retains all of its config in files (there's no database dependency). That seems to be the best route, as it's versioned and easy to maintain. Any thoughts on this would be well-appreciated.
Thanks!
@ameir the DB config is the source of truth, but due to the nature of WP, the advanced-cache.php is required to be a file. It's included before WP has it's DB layer and includes in place to save resources.
The only bypass I could see is to manually query the DB and get the configuration but I do not really wish to do that unless it's unavoidable.
To answer your question: the configuration gets saved in the DB immediately. Once the "save settings" is pushed, a function gets triggered (
deploy_advanced_cache()
) which generated the config.The settings getter/setter is in the wp-common/plugin_abstract class, which is not a true abstract but a common class for my plugins.
+1 to this, we currently get our memcached endpoints from provisioning time, and save them in a /config/ dir for later inclusion. Would be great if the advanced-cache.php file would remain static.
I think that if we ship a default configuration with the plugin and use advanced-cache.php as the source of record, it'd be ideal, and I don't think it'd be too hard to implement. I can give it a shot in the next week or so in the branch I'm developing against at https://github.com/ameir/wp-ffpc/tree/touchups .
Question for @petermolnar, if there is a difference between the config data in advanced-cache.php, and the config data in the db, what happens?
@drewsymo automatically: nothing. On settings save, the db data will overwrite the advanced-cache.php.
The problem is that in the advanced-cache.php I have no WP DB access at all. If I want a DB entry, I'd need manual queries - not that it's not doable but it's ugly and could be slow.
So if the advanced-cache file is immutable, e.g. cannot be written to, will the settings in advanced-cache.php still be used?
For instance, if I set the 'hosts' key in the advanced-cache.php file to my Memcached endpoints, will these be used for the Memcached backends? Even if the options page reflects that it's using 127.0.0.1 (or whatever the default this)?
The problem is that I've set pretty strict permissions, only /uploads/** is writable by the web group. I don't really mind if the settings page reflects the wrong settings for the time being.
EDIT: Just realised that this would bork the other options, like flushing the cache.
In our case, we're using and deploying advanced-cache.php across all of our hosts, and are completely disregarding the UI for settings. So long as changes aren't made in the UI, advanced-cache.php will remain correct. The UI will report different settings from those in the config file (unless you save anything in the UI, that is). We flush the cache via an external process, so I haven't checked whether doing a cache flush via the UI would overwrite advanced-cache.php.
I see. I'm wondering if a solution for the interim would be to:
From what I can tell, no writes are made to advanced-cache.php when clearing the cache.
Regardless, still feels a bit messy. @ameir, let me know when you're closer to making advanced-cache.php or similar the SSOT. Would be interesting to see how it works.
@drewsymo the ffpc config has to be included in the wp-content/advanced-cache.php file; it does not really matter where it's coming from, but at that step, most of the WordPress functions is not available yet. ( That is why it's fast ).
See:
What could be a possible step is to store the values in the backend(s) but with that, a flush could remove the settings ( external flush, php-fpm restart for example ) and we're without settings again.
Another approach could be to handcraft an SQL and manually look up the settings in the DB in case there is no config there is a config option forcing to do this.
I'll try to test this later solution although I welcome any code ideas.
Maybe I'm just stubborn, but I still like my original idea, which is to let WP-FFPC do it's thing as normal, but allow "advanced users" like ourselves to access the
global_config
array withindeploy_advanced_cache()
. Right now, without a hook there's no way to do this. I still feel advanced deployment (across multiple servers) should not be part of the plugin code. All we need is a door (by adding a hook), and we should be able to build this. Then we can write endpoints or deploy updates however we see fit.For my use case: take the PHP generated by
deploy_advanced_cache()
and stash it in the database. Then run a cron script checking this against the actual file. If there is a mismatch, or anything needs changing, update the actual advanced cache file with the database's stored code. Simple, and no performance hit.I think the way I phrased this in the original quote might have been confusing. I was not suggesting a static call at all, I was just identifying
deploy_advanced_cache
as a method of classWP_FFPC
. I wasn't talking about calling this directly. Only want a hook.The manual flush--how do you know that you're flushing the correct server? This doesn't solve the problem of knowing when to invalidate cache. This is possible of course, but we might be chasing our tail again.
The SQL--how do you keep from slowing down every page request? I see that as a big drawback, especially because this plugin is most effective when there's a DB bottleneck.
@haroldkyle, I like this idea. For some reason I was off on my own little tangent.
So, is the best method to do an
apply_filters
on the $global_config / $options array? I'll try this out and see how it goes.@haroldkyle I could put in a WP cron hook to regenerate the advanced-cache.php, would that help?
@petermolnar Well...I considered WP cron but doesn't that solution rely on WP loading? If the server is caching pages well (without hitting WP at all), I thought this scheduled job would never execute. That's I thought a system cron would be preferential.
@drewsymo I went back and forth on PHP array vs. PHP code. I would think we could add
apply_filters
for both...why not? In my instance, I would probably prefer the code so my system script can be as dumb as possible. On a side note, I'm curious how you assign memcache endpoints at provisioning--is that through AWS or do you have your own solution?I use a deployment for heroku that has advanced-cache.php in place and it uses getenv to get connection to memcached. Might this help?
http://12factor.net/config
I updated my fork (which I hope I can merge via PR sometime) to not need the database at all. Any non-default settings are loaded via a config.php file. In short, you can make changes to config.php, and include it in your deployment process. If it is not present, you will inherit the defaults. Upon each save, a new config.php file will be generated with the live options. advanced-cache.php now acts as a bootstrap to the plugin. The database is not part of this process at all.
You can view my updates at https://github.com/ameir/wp-ffpc/tree/touchups . It'd be great if I could get others to test. If you have any questions or concerns, please let me know. Once my branch is quality tested, I'll try to have it merged. It's currently running on a pretty busy site that gets a few million hits per day. So far so good.
I'm running on Heroku and I really need to be able to store the settings in a config file in which, for example, I can then read certain settings from
$_ENV
(like @larsschenk).For example another plugin I use [https://github.com/deliciousbrains/wp-amazon-web-services] allows you to define your AWS credentials as constants in wp-config (which I can grab from
$_ENV
), and when it detects this the relevant admin setting UI fields are disabled. I think this approach would work well here.I'm keen to get this working and contribute / help test. I could implement this but given work has already been done it would be sensible to continue this rather than me fork the main repo.
@ameir are you close to submitting a PR for your updates?
Where did this get to? I have just started using wp-ffpc after getting frustrated with the bloat and unnecessary complexity of other well known cache plugins.
I too have multiple EC2 servers running behind a load balancer. They store changes in a seperate memcache server from where Nginx on each server looks direct and on a miss calls up wordpres so I return the page etc. I push changes out via git with post receive rules to deploy in production. All the production servers have the same basic builds. I use an S3 plugin to push all uploads to AWS S3 so the servers are stateless. This setup works very well and I can spin servers up on demand and they work as expected, other than one concern. I'm not sure they are all caching.
I maintain the advanced cache settings on my development build from where I push the changes out.
I have activated the plugin done an initial save of the settings. Then I have pushed an updated advanced-cache.php up to the production servers with the settings I want. I recently (today) decided to start using the sha1 hash for keys.
My question is this. After activating the plugin and an initial save does the plugin work completely off settings it finds in the advanced cache file on a server in normal operations? I.e can I purely update settings via pushing out a new advanced cache file to my production servers?
My suggestions would be to save the config settings in a seperate file within your plugin directory. Something named wp-ffpc-config.php say. Then when you activate your plugin it overwrites advanced cache with an include for that file , followed by say wp-user-settings.php , followed by an include for your acache file.
Any user overrides can be stored in the second file your plugin never overrides but always looks for. If user overrides file is present then the admin screens just say so, and don't allow the settings to be updated. If user settings file not present, then it's business as usual.