TYPO free

home

fighting for TYPO free code

TYPO3 housekeeping

25 Jun 2008

When you run a TYPO3 installation for several months you tend to create and delete a lot of pages and a lot of page content. Relations between elements are created and severed. When pages and records are deleted in TYPO3, they are not really deleted from the database. The 'deleted' field in the table is merely set to 1. So over time your table will accumulate a lot of deleted records.

It can also happen that you remove a page that is pointed to by another page. You could have seen this by looking at the references to a page, but let's say yo did not and just dropped the page. The page that was pointing to it will now have a broken reference. It can also happen that you have templates or pages that no other page or template uses (orphans). There may also be "soft references" from eg. typolinks (<link 123>...</link>) or file references in a TypoScript template (something.file = fileadmin/template/miss_me.jpg) which are missing. Those cannot be automatically repaired but the cleanup script can warn you about these broken references if they exist so you can fix them.

All these records should be inspected once in a while to see what can safely be removed. The result will hopefully be a smaller, lighter and hopefully faster database.

TYPO3 comes with a tool to help you with these housekeeping tasks. It's hidden away in: typo3/sysext/lowlevel. There is some documentation in a file called HOWTO_clean_up_TYPO3_installations.txt in that directory. Other than that there is very little documentation available on the net. So let's talk about the different tools in the TYPO3 cleaning set an how to use them effectively.

Near the top of the HOWTO there is a big red warning sign saying: THERE IS ABSOLUTELY NO WARRANTY associated with this script! It is completely on your OWN RISK that you run it. It may cause accidential data loss due to software bugs or circumstances that it does not know about yet - or data loss might happen due to misuse!

So . . . proceed at your own risk. ;-)

But seriously, to repeat:

  • Backups are good! (sql + files)
  • Before running with the --AUTOFIX option ALWAYS make sure to add the parameter "--dryrun" to see what would be fixed.
  • Also, NEVER BYPASS the REFERENCE INDEX CHECK if --AUTOFIX is used for those tools which require a clean reference index.
  • It could be a good idea to run a myisamchk on your database just to make sure MySQL has everything pulled together right. Something like this will do: myisamchk [path_to_mysql_databases]/[database_name]/*.MYI -s -r

Oh, and two more things before we begin:

  1. Create a regular backend user with random password called _cli_lowlevel (this user will run as admin in the live workspace)
  2. If you are on a live server, prefix all the php commands with "nice -n 19" so you don't disturb the other processes

Pull up your favorite terminal and fire up the cleaner by invoking the cli_dispatch.phpsh script with the lowlevel_cleaner as option:

cd /path/to/typo3_site
php typo3/cli_dispatch.phpsh lowlevel_cleaner

This command will spit back the name, synopsis, description and the list of tools in our toolbox:

NAME:
lowlevel_cleaner -- Analysis and clean-up tools for TYPO3 installations

SYNOPSIS:
toolkey [-s] [--silent] [-ss] [-r] [-v level] [--refindex mode]
[--AUTOFIX [testName]] [--dryrun] [--YES] [-i] [--filterRegex expr]
[--showhowto]

DESCRIPTION:
Dispatches to various analysis and clean-up tools which can plug into
the API of this script. Typically you can run tests that will take
longer than the usual max execution time of PHP. Such tasks could be
checking for orphan records in the page tree or flushing all published
versions in the system. For the complete list of options, please
explore each of the 'toolkey' keywords below:

missing_files
missing_relations
double_files
rte_images
lost_files
orphan_records
deleted
versions
cleanflexform
syslog
tx_templavoila_unusedce

Nice tools! Let's run a test command with the following options:

-s                      Silent operation, will only output errors and
                        important messages.
-r                      Execute this tool, otherwise help is shown

This will give us an idea of what the cleaner does.

php typo3/cli_dispatch.phpsh lowlevel_cleaner deleted -s -r

This gives us some output:

---------------------------------------------
[deleted]
Index of deleted records [INFO]
---------------------------------------------
Explanation: These are records from the page tree having the deleted-flag set. The
--AUTOFIX option will flush them completely!

Array
(
[fe_groups] => Array
(
[1] => 1
[2] => 2
[3] => 3
)

[fe_users] => Array
(
[1] => 1
[2] => 2
)

[pages] => Array
(
[4] => 4
[5] => 5
[6] => 6
[7] => 7
)
)

Ok, we know what is idling in the db tables and we are positive we don't nee those records any more so we run:

php typo3/cli_dispatch.phpsh lowlevel_cleaner deleted -s -r --AUTOFIX

We'll see the same output but now it ends with the line:

NOW Running --AUTOFIX on result. OK? (Yes/No + return):

If you are brave; type Yes and hit return. The cleaner does it's work and cleans our database. If you are not so brave; re-run the command with the --dryrun option and type Yes and enter to see what will happen.

Flushing deleted records from table "fe_groups":
    Flushing record "be_users:1": DONE
    Flushing record "be_users:2": DONE
    Flushing record "be_users:3": DONE
    ...

That's pretty neat! The howto suggests an optimal order of running the tools:

  • orphan_records - As a beginning, get all orphaned records out of the system since you probably want to. Since orphan records may keep some missing relations from being detected it's a good idea to get rid of them immediately.
  • versions - Flush all published versions now if you like. Published versions may also keep references to records which could affect other tests, hence do it now if you want to.
  • tx_templavoila_unusedce - (Assumes usage of "TemplaVoila" extension!)
    • This should be done AFTER flushing published versions (since versions could reference elements that might be safe to remove)
    • This should be done BEFORE flushing deleted versions (since this tool will create new deleted records), given that you want to completely flush them of course.
    • You should run it over again until there remains no more unused elements. You need to do this because deleting elements might generate new unused elements if the now-deleted elements had references.
  • double_files - Fix any files referenced twice or more before you delete records (which could potentially delete a file that is referenced by another file).
  • deleted - Flush deleted records. As a rule of thumb, tools that create deleted records should be run before this one so the deleted records they create are also flushed (if you like to of course)
  • missing_relations
  • cleanflexform - After the "deleted" tool since we cannot clean-up deleted records and to make sure nothing unimportant is cleaned up
  • rte_images - Will be affected by flushed deleted records, versions and orphans so must be run after any of those tests.

Two tools can be used any time:

  • missing_files
  • lost_files

So, go practice a bit. More to come in TYPO3 housekeeping part 2.

Steffen 26 Jun 2008, 22:21
Hi Michiel,
this article looks so handy to me atm. I was just starting to migrate some websites to a new server and looking for some information about cleanup TYPO3.
Thanks a lot.

Steffen
Domi 19 Aug 2011, 08:09
Hi Michiel,

I tried your sh TYPO3cleaner.sh script, but it just gives me error:
TYPO3cleaner.sh: 3:  : not found
TYPO3cleaner.sh: 7:  : not found
TYPO3cleaner.sh: 8:  : not found
TYPO3cleaner.sh: 9:  : not found
TYPO3cleaner.sh: 22:  : not found
TYPO3cleaner.sh: 23:  : not found
TYPO3cleaner.sh: 24:  : not found
TYPO3cleaner.sh: 32:  : not found
TYPO3cleaner.sh: 32:  : not found
TYPO3cleaner.sh: 33:  : not found
TYPO3cleaner.sh: 34:  : not found
TYPO3cleaner.sh: 35:  : not found
TYPO3cleaner.sh: 39: Syntax error: ")" unexpected (expecting "done")

There are problems witch empty lines and the other Syntax error I couldn't fix.

But very nice tutorial!
Domi
Michiel 19 Aug 2011, 09:38
Hi Domi,

Please use the last script from the article https://www.typofree.org/article/archive/2008/june/title/typo3-housekeeping-part-2/.

It should just work (tm).

Otherwise please contact me by mail to sort it out. Thx.

Michiel
Commenting is closed for this item