Reputation Top 1%
Fabio Ros
10 Snippets  (96th place)
1 Channel
4 Channels
Jan 8, 2017
Last Visit
Dec 25, 2016
242 points  (124th place)
Junior Code Generator
Serious Code Generator
Serious Popular Coder
Junior Autobiographer
Senior Autobiographer
Junior Famous Coder

Recent Snippets See all snippets by fabioros

public by fabior created Nov 26, 2016  931  0  3  0

Magento 1 Universal Ignore - optimised for security in various environments

Magento 1 Universal Ignore - optimised for security in various environments: .gitignore
# Magento 1 Universal ignore V:1 ~ FROSIT.nl
# Project: ...

# IDE&OS files

# Dynamic data which shouldn't be in version control

# Explicitly mark as "do not exclude/ignore these from GIT repository"


# Developer, project and security precautions

# Hypernode specfic

# Developer folder to store and secure stuff temporarily

# Project related
# ...somefile.ext

# Bulk remaining auto appended ignores below if necessary ===


public by fabior created Nov 23, 2016  1469  0  3  0

Magelock shows your indexes and lock files via a semi-secure one-page easy-to-use and discard way

Magelock shows your indexes and lock files via a semi-secure one-page easy-to-use and discard way: Locks.php
 * == MageLock ==
 * ~ Shows your Magento indexes' lock status.
 * @description This file shows your index statuses and was meant to show when lock's became active or inactive.
 * @note it requires the user to be logged in the backend for output to show, however this is not sufficient for proper security
 * == How To ==
 * Drop this file anywhere where it can reach app/Mage.php
 * Login the backend and return to the file
 * Additionally, change the params for dev
 * == To Do
 * @todo add angular for refreshless gui
 * == License
 * @author      Fabio Ros <f.ros@frosit.nl>
 * @copyright   Copyright (c) 2016 Fabio Ros - FROSIT
 * @license     https://opensource.org/licenses/Apache-2.0 Apache License, Version 2.0
 * @see @license @bottom-of-the-page

 * == Dev params
ini_set('display_errors', 1);
ini_set('max_execution_time', -1);
// EOD

 * === Should reach Mage.php
$mage = __DIR__.DIRECTORY_SEPARATOR.'app/Mage.php';

if (file_exists($mage)) {
    require_once $mage;
} else {
    echo 'Could not find mage file at path: '.$mage;

\Mage::app('default'); // change if needed

// be authorized / logged in
\Mage::getSingleton('core/session', array('name' => 'adminhtml'));
$session = Mage::getSingleton('admin/session');

if (!$session->isLoggedIn()) {
    echo 'Unauthorized';

// initializing values
 * Class MageLock
class MageLock
     * @var array|int|null|string
    protected $url = null;
     * @var Mage_Index_Model_Indexer $_indexes
    protected $_indexes = null;
     * @var Mage_Admin_Model_Session|Mage_Core_Model_Abstract
    protected $session;

     * MageLock constructor.
    public function __construct()
        $this->session = Mage::getSingleton('admin/session');
        if (!$this->session->isLoggedIn()) {
            echo 'Please login the backend to get access to this file';

        $this->indexes = Mage::getSingleton('index/indexer')->getProcessesCollection()->load();
        $this->url = $_SERVER['PHP_SELF'];

     * @return array|string
    public function getUrl()
        if (null === $this->url) {
            $this->url = $_SERVER['PHP_SELF'];

        return $this->url;

     * @param bool $locked
     * @return array|null|Varien_Data_Collection_Db
    public function getIndexes($locked = false)
        if (null !== $this->_indexes) {
            return $this->_indexes;
        $locks = [];
        $indexes = $this->_indexes;
        foreach ($indexes as $index) {
            if ($locked) {
                if ($index->isLocked()) {
                    $locks[$index->getIndexerCode()] = $index->getData();
            } else {
                $locks[$index->getIndexerCode()] = $index->getData();

        return $locks;

     * @param null $indexes
     * @param bool $locked
     * @return array
    public function getAsTable($indexes = null, $locked = false)
        if (null === $indexes) {
            $indexes = $this->indexes;

        $table = array('rows' => array(),);
        $i = 0;
        foreach ($indexes as $index) {
            $lock = $index->isLocked() ? 'true' : 'false';
            $index = $index->getData();

            if ($i === 0) {
                $headers = array_keys($index);
                $headers[] = 'lock';
                $table['headers'] = $headers;

            $rows = array_values($index);
            $rows[] = $lock;
            $table['rows'][] = $rows;

        return $table;

     * @param $text
     * @return mixed
    public function convertToTile($text)
        $pieces = str_replace('_', ' ', $text);

        return uc_words($pieces);

 * Initials for GUI
$mageLock = new MageLock();
$tableData = $mageLock->getAsTable(null, true);

<!doctype html>
<html lang="en">
    <meta charset="utf-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <title>Index Locks</title>
    <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0-alpha.5/css/bootstrap.min.css"
          integrity="sha384-AysaV+vQoT3kOAXZkl02PThvDr8HYKPZhNT5h/CXfBThSRXQ6jW5DO2ekP5ViFdi" crossorigin="anonymous">
   * @todo implement Angular for usage without page refreshes
    <script src="https://ajax.googleapis.com/ajax/libs/angularjs/1.5.8/angular.min.js"></script>
        body {
            padding-top: 5rem;

        .starter-template {
            padding: 3rem 1.5rem;
            text-align: center;

        footer {
            text-align: center;
<body class="bg-faded">
<nav id="topNav" class="navbar navbar-default navbar-fixed-top navbar-dark bg-inverse">
    <div class="container">
        <button class="navbar-toggler hidden-md-up pull-right" type="button" data-toggle="collapse"
        <a class="navbar-brand page-scroll" href="#">FROSIT</a>
        <div class="collapse navbar-toggleable-sm" id="collapsingNavbar">
            <ul class="nav navbar-nav">
                <li class="nav-item">
                    <a class="nav-link page-scroll" href="#">Awesome</a>

<div class="container">
    <div class="starter-template">
        <h1>Magento Locked indexes
            <small>- FROSIT</small>
        <p class="lead">Get an overview of locked indexes.</p>

</div><!-- /.container -->

<div class="container">
    <div class="row">
        <div class="col-md-12">

            <table class="table">
                    <?php foreach ($tableData['headers'] as $header): ?>
                            <?php echo $mageLock->convertToTile($header) ?>
                    <?php endforeach; ?>
                <?php foreach ($tableData['rows'] as $row): ?>
                        <?php foreach ($row as $item): ?>
                            <td><?php echo $item ?></td>
                        <? Endforeach; ?>
                <? Endforeach; ?>
<footer id="footer">
    <div class="container">
        <span class="text-muted small"><a href="http://frosit.nl">FROSIT</a> ©2015-2016</span>
<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.1.1/jquery.min.js"
<script src="https://cdnjs.cloudflare.com/ajax/libs/tether/1.3.7/js/tether.min.js"
<script src="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0-alpha.5/js/bootstrap.min.js"
Apache License
                           Version 2.0, January 2004


   1. Definitions.

"License" shall mean the terms and conditions for use, reproduction,
      and distribution as defined by Sections 1 through 9 of this document.

"Licensor" shall mean the copyright owner or entity authorized by
      the copyright owner that is granting the License.

"Legal Entity" shall mean the union of the acting entity and all
      other entities that control, are controlled by, or are under common
      control with that entity. For the purposes of this definition,
      "control" means (i) the power, direct or indirect, to cause the
      direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
      outstanding shares, or (iii) beneficial ownership of such entity.

"You" (or "Your") shall mean an individual or Legal Entity
      exercising permissions granted by this License.

"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
      source, and configuration files.

"Object" form shall mean any form resulting from mechanical
      transformation or translation of a Source form, including but
      not limited to compiled object code, generated documentation,
      and conversions to other media types.

"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
      copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).

      "Derivative Works" shall mean any work, whether in Source or Object
      form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
      represent, as a whole, an original work of authorship. For the purposes
      of this License, Derivative Works shall not include works that remain
      separable from, or merely link (or bind by name) to the interfaces of,
      the Work and Derivative Works thereof.

"Contribution" shall mean any work of authorship, including
      the original version of the Work and any modifications or additions
      to that Work or Derivative Works thereof, that is intentionally
      submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
      the copyright owner. For the purposes of this definition, "submitted"
      means any form of electronic, verbal, or written communication sent
      to the Licensor or its representatives, including but not limited to
      communication on electronic mailing lists, source code control systems,
      and issue tracking systems that are managed by, or on behalf of, the
      Licensor for the purpose of discussing and improving the Work, but
      excluding communication that is conspicuously marked or otherwise
      designated in writing by the copyright owner as "Not a Contribution."

      "Contributor" shall mean Licensor and any individual or Legal Entity
      on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.

2. Grant of Copyright License. Subject to the terms and conditions of
      this License, each Contributor hereby grants to You a perpetual,
      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
      copyright license to reproduce, prepare Derivative Works of,
      publicly display, publicly perform, sublicense, and distribute the
      Work and such Derivative Works in Source or Object form.

3. Grant of Patent License. Subject to the terms and conditions of
      this License, each Contributor hereby grants to You a perpetual,
      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
      use, offer to sell, sell, import, and otherwise transfer the Work,
      where such license applies only to those patent claims licensable
      by such Contributor that are necessarily infringed by their
      Contribution(s) alone or by combination of their Contribution(s)
      with the Work to which such Contribution(s) was submitted. If You
      institute patent litigation against any entity (including a
      cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
      granted to You under this License for that Work shall terminate
      as of the date such litigation is filed.

4. Redistribution. You may reproduce and distribute copies of the
      Work or Derivative Works thereof in any medium, with or without
      modifications, and in Source or Object form, provided that You
      meet the following conditions:

      (a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and

      (b) You must cause any modified files to carry prominent notices
          stating that You changed the files; and

      (c) You must retain, in the Source form of any Derivative Works
          that You distribute, all copyright, patent, trademark, and
          attribution notices from the Source form of the Work,
          excluding those notices that do not pertain to any part of
          the Derivative Works; and

      (d) If the Work includes a "NOTICE" text file as part of its
          distribution, then any Derivative Works that You distribute must
          include a readable copy of the attribution notices contained
          within such NOTICE file, excluding those notices that do not
          pertain to any part of the Derivative Works, in at least one
          of the following places: within a NOTICE text file distributed
          as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
          within a display generated by the Derivative Works, if and
          wherever such third-party notices normally appear. The contents
          of the NOTICE file are for informational purposes only and
          do not modify the License. You may add Your own attribution
          notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
          that such additional attribution notices cannot be construed
          as modifying the License.

You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
      for use, reproduction, or distribution of Your modifications, or
      for any such Derivative Works as a whole, provided Your use,
    reproduction, and distribution of the Work otherwise complies with
      the conditions stated in this License.

5. Submission of Contributions. Unless You explicitly state otherwise,
      any Contribution intentionally submitted for inclusion in the Work
      by You to the Licensor shall be under the terms and conditions of
      this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
      the terms of any separate license agreement you may have executed
      with Licensor regarding such Contributions.

6. Trademarks. This License does not grant permission to use the trade
      names, trademarks, service marks, or product names of the Licensor,
      except as required for reasonable and customary use in describing the
      origin of the Work and reproducing the content of the NOTICE file.

7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
      Contributor provides its Contributions) on an "AS IS" BASIS,
implied, including, without limitation, any warranties or conditions

 * PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
      risks associated with Your exercise of permissions under this License.

8. Limitation of Liability. In no event and under no legal theory,
      whether in tort (including negligence), contract, or otherwise,
      unless required by applicable law (such as deliberate and grossly
      negligent acts) or agreed to in writing, shall any Contributor be
      liable to You for damages, including any direct, indirect, special,
                        incidental, or consequential damages of any character arising as a
      result of this License or out of the use or inability to use the
      Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
      other commercial damages or losses), even if such Contributor
      has been advised of the possibility of such damages.

9. Accepting Warranty or Additional Liability. While redistributing
      the Work or Derivative Works thereof, You may choose to offer,
      and charge a fee for, acceptance of support, warranty, indemnity,
      or other liability obligations and/or rights consistent with this
      License. However, in accepting such obligations, You may act only
      on Your own behalf and on Your sole responsibility, not on behalf
      of any other Contributor, and only if You agree to indemnify,
      defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
      of your accepting any such warranty or additional liability.


   APPENDIX: How to apply the Apache License to your work.

To apply the Apache License to your work, attach the following
      boilerplate notice, with the fields enclosed by brackets "{}"
      replaced with your own identifying information. (Don't include
      the brackets!)  The text should be enclosed in the appropriate
      comment syntax for the file format. We also recommend that a
      file or class name and description of purpose be included on the
      same "printed page" as the copyright notice for easier
      identification within third-party archives.

   Copyright {yyyy} {name of copyright owner}

   Licensed under the Apache License, Version 2.0 (the "License");
   you may not use this file except in compliance with the License.
   You may obtain a copy of the License at


   Unless required by applicable law or agreed to in writing, software
   distributed under the License is distributed on an "AS IS" BASIS,
   See the License for the specific language governing permissions and
   limitations under the License.


public by fabior created Oct 3, 2016  2051  7  3  0

Quick reference for rewritetoolset

Quick reference for rewritetoolset: quick-reference.md
Commands for hypernode

__Quick note on command options__

Due to the earlier analysis phase of this set. There are some option embedded which are redundant to some commands. These options will show available, but won't do anything at some newer commands. They are mostly available to analysis and benchmark commands.

_Sometimes redundant options_

* --save (sometimes generates a HTML report)
* --share-statistics (disabled by default)
* --log-statistics (mostly generates a JSON file on var/rewrite_tools/stats)
* --store (mostly available)

# Getting started real quick

### 1. Analysis

`magerun rewrites:analysis:totals --store all`

`magerun rewrites:analysis:top --store all`

### 2. Measuring (optional)

`rewrites:benchmark:resolve-urls --store all`

`magerun rewrites:benchmark:indexer --limit 1`

### 3. Heavy cleaning

__Safe cleaning__

`rewrites:clean:disabled --store all`

`rewrites:clean:older-than 90` 

__Risky cleaning__

`magerun rewrites:clean:yolo`

### 4. Permanent fix

`magerun rewrites:fix:products` 


more in depth information.

# 1. Analyse the problem

First we need some indication on how big the problem is. 2 commands are very suitable for that:

__Get duplicate totals__

Easier to immediately check for all stores

`magerun rewrites:analysis:totals --store all`

If you're hitting the 1 million of dupes, it's critical. +90% percentages are easily reached.
@todo explain when to continue

__Get top duplicated products__

We can see which products or categories cause the problem by running the following command.

`magerun rewrites:analysis:top --store all`

If there are a couple of products or categories that cause most of the duplicates, it could be fixed easily and even manually fixing the url keys can be an option over using experimental and complex fix commands.

## 1-1 What is the impact on loading times

We can measure the impact of the problem by benchmarking.
There are benchmark commands for indexing times, url resolvement times and site performance. The first 2 are most usable but may take a while if the problem is real.

__URL resolve times__

This basically generates a sitemap. Magento has the process all duplicates to find the actual ones, if your indexes are not up-to-date (probably not), this will take a while. If your hitting the millions of dupes, skip it. it takes too long.

`rewrites:benchmark:resolve-urls --store all`

__note:__ this command also fixes some outdated indexes, the second time will finish almost instant due to up to date indexes.

__Indexer times__

The following command runs a full reindex of the _catalog_url_ index and outputs the runtime.
This one is set to 10 runs by default, but if you just want to know the time, set the limit to 1. That is enough information on how long it runs, and how money duplicates are created on each run.

`magerun rewrites:benchmark:indexer --limit 1`

# 2. Decide the next course of action

Depending on earlier results, we know the impact and scale of the problem.
Another thing we need to know, is the store's state. Is it a new shop, high volume / traffic show, SEO scores etc.
The hard part about fixing this problem is maintaining SEO score, while guaranteeing uptime's.

Ask yourself these questions:

_Is it a problem to lose all SEO scores?_

* Yes: Start with building the whitelist
* No: Lucky you

_Is it a problem to go offline or respond really slow for a relative amount of time? (tens of minutes to maybe hours)_

* Yes: Continue with a test  / development setup
* No: Again, lucky you

_Was there enough time to create a fresh can of coffee while running the analysis commands?_

* Yes: Start with some heavy cleaning
* No: Go right up to fixing

## Do some heavy cleaning

In some cases, the indexes are so clogged that it's barely workable. You probably have tons of rewrites which aren't used anyway. We have commands to clean those out.

__Important note__

These commands have a dry-run options (--dry-run). Use them to check the response first.

__Disabled products and store views__

Most stores have store views and or products which are disabled. We don't need those rewrites. Although they will be recreated if we don't fix them. Wiping them out makes the database somewhat more workable. 

`rewrites:clean:disabled --store all`

__Old rewrites__

The clean:older-than command removes all rewrites which are older than x days from now. Most stores with big rewrite problems are building them up for several years. After a couple of months, the old ones shouldn't be indexed by google anymore. There's no mechanism to check for indexed URL's yet, but this is possible if this toolset proves worthy. The whitelist commands have options to whitelist url's by CSV (google analytics), access logs or the visitor log with another time x days option. They are not available to this clean:older-than command, yet.

Run the following command:

`rewrites:clean:older-than days` 

where days is the number of days in the past you want to preserve rewrites for. The --store option is available, but i prefer to set it to --store all
The --dry-run options works too.


There are some scenario's where a shop lost all their scores, income, and everything is pretty messed up and clogged etc. When there's nothing you need to preserve, it's time to start fresh.
The clean yolo command _removes every duplicate rewrite_ without checking other things like times, whitelists, disabled statuses or whatsoever. You still have to fix the duplicate keys but the store wont be blocking that much. You only live once right, cliches are there for the ones who have nothing to care about.

`magerun rewrites:clean:yolo`

This command has the --dry-run options available

## Permanent fixing

The whitelist commands are not yet integrated with permanent fixing. therefore only 1 permanent fix command is available. But i's a general good fix.

`magerun rewrites:fix:products` 

* Start with the `--dry-run` option
* Optionally specify a different suffix if urls seem out of shape `--new-suffix`

## Building the whitelist (unfinished)

In this stage, it is required to keep your SEO scores. We can make this more safe by creating a whitelist of rewrite urls by adding URLs from different sources. The goal is to whitelist URL's that we're recently visited and are probably indexed by google. Sources are Google analytics ar any other website statistics tool or service (CSV), Magento's visitor log table and server acces logs. 

__note__ Building whitelists is mostly finished, using them not yet. Testing the whitelist build-up would be appreciated. The implementation does not require a lot of time and will be done a.s.a.p.

Start by adding URL's from sources available to you.

### Adding sources to the whitelist databases

We use JSON based database for this. Each command builds a seperate JSON database which later on will be processed into a master whitelist. This master whitelist is then used to backup rewrites or ignore the removal of rewrites.

#### From access logs

This command currently only works for hypernode. Parsing all those access logs requires quite a lot of memory. The command is aware of it's memory usage and will automatically trigger garbage collecting to lower it's memory usage. Problems could come up in high traffic stores.

`magerun   rewrites:log:parse --file="/var/log/nginx/access.log*"`

#### From visitor logs

This command processer Magento's visitor log table into a whitelist database. Rewrites older than x days will be filtered out. By default this is set to 60 days.

` magerun  rewrites:url:visitor --max-age 90`

#### From CSV

This is the best and most safe option of preserving Google SEO scores as CSV from Google analytics can be added to the whitelist. Specify a path to the CSV and a column to take URL's from.

`magerun rewrites:url:csv --csv path/to/csv.csv --column urlcolumn`

### Building the whitelist

When all whitelist sources are converted to whitelist json databases, it is time to process these into one master whitelist. All sources are combined and cut up into segments. For example:

_url_ : some_product_url_duplicatevalue.html
_segment_ : some_product_url

Each segment is then queried against the rewrites database and each result of this segment query is filtered on a max-age and matched back against the combined sources of whitelists. _each match_ is then added to the master whitelist. This dramatically reduces the size of the master whitelist and load on the database. The extra filter makes sure there are no redundant rewrites added to the master list. If all sources are added correctly, this is a strong safeguard on maintaining valuable URL's. The master whitelist will be used to backup everything in the future so rewrites could be wiped and recreated after fixing the keys. Unfortunately, this is a work in progress. But testing the buildup is gladly appreciated.

This concept is a complex one, i'm happy to explain it in-dept.


`magerun rewrites:url:whitelist`


public by fabior created Aug 25, 2016  974  2  3  0

Backup all databases in seperate archives

Backup all databases in seperate archives: backup-databases.sh
# List all databases, backup them seperately gzipped

now="$(date +'%d-%m-%Y')"
DATABASES=`mysql -u ${USER} -p${PASSWORD} -h ${HOST} -e "SHOW DATABASES;" | tr -d "| " | grep -v Database`

for db in ${DATABASES}; do
    if [[ ${db} != "information_schema" ]] && [[ ${db} != "performance_schema" ]] && [[ ${db} != "mysql" ]] && [[ ${db} != _* ]] ; then
        echo "Dumping database: "${db}
        mysqldump -u ${USER} -p${PASSWORD} -h dbint059803 ${db} | gzip -v > "backups/db/${now}_${db}.sql.gz"


public by fabior created Jul 5, 2016  918  0  3  0

Usefull Magento Dev Commands

Usefull Magento Dev Commands: commands.md
Usefull Magento dev commands

This gist contains some of my Magento development commands.

# Logs

Get list of 100 most frequent log errors

``` bash
cut -d ' ' -f 2- system.log | sort | uniq -c | sort -n -k 1 | tail -100