1. 09 9月, 2019 1 次提交
    • K
      Improve performance and memory usage of project export · 0e56c1e7
      Kamil Trzciński 提交于
      ActiveModel::Serialization is simple in that it recursively calls
      `as_json` on each object to serialize everything. However, for a model
      like a Project, this can generate a query for every single association,
      which can add up to tens of thousands of queries and lead to memory
      bloat.
      
      To improve this, we can do several things:
      
      1. We use `tree:` and `preload:` to automatically generate
         a list of all preloads that could be used to serialize
         objects in bulk.
      
      2. We observe that a single project has many issues, merge requests,
         etc. Instead of serializing everything at once, which could lead to
         database timeouts and high memory usage, we take each top-level
         association and serialize the data in batches.
      
      For example, we serialize the first 100 issues and preload all of
      their associated events, notes, etc. before moving onto the next
      batch. When we're done, we serialize merge requests in the same way.
      We repeat this pattern for the remaining associations specified in
      import_export.yml.
      0e56c1e7
  2. 06 9月, 2019 1 次提交
    • K
      Normalize import_export structure · 0eeadb2d
      Kamil Trzciński 提交于
      This brings a significant refactor to how
      we handle `import_export.yml`, merge it with EE
      and how we handle that for reader and saver.
      
      This is meant to simplify the code, and remove
      a ton of conditions to handle different models
      of the structure.
      
      This is also meant to prepare the structure
      to extend it much easier, like adding `preload:`
      or additional object types when needed.
      
      This does not change the behavior of import/export,
      rather unifies and simplifies the current implementation.
      0eeadb2d
  3. 17 6月, 2019 1 次提交
    • Y
      Backport the EE schema and migrations to CE · 8469f59d
      Yorick Peterse 提交于
      This backports all EE schema changes to CE, including EE migrations,
      ensuring both use the same schema.
      
      == Updated tests
      
      A spec related to ghost and support bot users had to be modified to make
      it pass. The spec in question assumes that the "support_bot" column
      exists when defining the spec. In the single codebase setup this is not
      the case, as the column is backported in a later migration. Any attempt
      to use a different schema version or use of "around" blocks to
      conditionally disable specs won't help, as reverting the backport
      migration would also drop the "support_bot" column. Removing the
      "support_bot" tests entirely appears to be the only solution.
      
      We also need to update some foreign key tests now that we have
      backported the EE columns. Fortunately, these changes are very minor.
      
      == Backporting migrations
      
      This commit moves EE specific migrations (except those for the Geo
      tracking database) and related files to CE, and also removes any traces
      of the ee/db directory.
      
      Some migrations had to be modified or removed, as they no longer work
      with the schema being backported. These migrations were all quite old,
      so we opted for removing them where modifying them would take too much
      time and effort.
      
      Some old migrations were modified in EE, while also existing in CE. In
      these cases we took the EE code, and in one case removed them entirely.
      It's not worth spending time trying to merge these changes somehow as we
      plan to remove old migrations around the release of 12.0, see
      https://gitlab.com/gitlab-org/gitlab-ce/issues/59177 for more details.
      8469f59d