Memory issues with the Magento 2.0 data migration tool

The Magento 2.x data migration tool works well, but sometimes has issues with very large tables on lower end servers that don’t have a ton of memory available.

We ran into this issue when migrating a client this morning, specifically in the code here:

vendor/magento/data-migration-tool/src/Migration/ResourceModel/Adapter/Mysql.php on line 161

Here was the associated code:

protected function insertMultiple($documentName, $records)
    {
        $bind = [];
        $values = [];
        $colNum = count($records[0]);
        $fields = array_keys($records[0]);
        foreach ($records as $record) {
            foreach ($record as $value) {
                $bind[] = $value;
            }
            $values[] = '(' . implode(',', array_fill(0, $colNum, '?')) . ')';
        }
        if ($values && $fields) {
            $insertSql = sprintf(
                'INSERT INTO %s (%s) VALUES %s',
                $documentName,
                sprintf('`%s`', implode('`,`', $fields)),
                implode(',', $values)
            );

            $statement = $this->resourceAdapter->getConnection()->prepare($insertSql);
            $statement->execute($bind);
        }
​
        return true;
    }

The error was happening when a lot of data was hitting the $statment->execute($bind);

So, we updated the function to run single inserts for each row (MySQL documentation warns us that this is much slower, but at least it works!):

protected function insertMultiple($documentName, $records)
    {

        $colNum = count($records[0]);
        $fields = array_keys($records[0]);
        foreach ($records as $record) {
            $row_bind = [];
            $row_values = [];
            foreach ($record as $value) {
                $row_bind[] = $value;
            }
            $row_values[] = '(' . implode(',', array_fill(0, $colNum, '?')) . ')';

            $insertSql = sprintf(
                'INSERT INTO %s (%s) VALUES %s',
                $documentName,
                sprintf('`%s`', implode('`,`', $fields)),
                implode(',', $row_values)
            );

            $statement = $this->resourceAdapter->getConnection()->prepare($insertSql);
            $statement->execute($row_bind);

        }
​
        return true;
    }

Are you having issues migrating your data to a Magento 2.x site? Contact Us and we will get you all squared away! We are Magento Certified and can tackle any issues you might have!