Archive of articles classified as' "git"

Back home

Quick Shell Function to Bootstrap a Gradle Groovy Project

2012/02/29

Gradle is a great build tool. It’s easy to download and install. If you’re on a mac and have homebrew, it’s as easy as:

brew install gradle

It’s very easy to use with a little experience, but I find having a good starting place to base your work from can help.

Here’s a quick function that I’ve got in my .zshrc to bootstrap a new groovy gradle project in the current directory (it should also work in in a .profile/.bash_profile/.bashrc).

function newgradle() {
    echo "Creating files for new gradle project"
 
    cat <<EOF>.gitignore
*.un~
*.iml
*.ipr
*.iws
build
.gradle
EOF
 
    cat <<EOF>build.gradle
apply plugin: 'groovy'
apply plugin: 'idea'
 
repositories {
    mavenCentral()
}
 
dependencies {
    groovy 'org.codehaus.groovy:groovy-all:1.8.6'
    groovy 'org.apache.ivy:ivy:2.2.0'
    testCompile 'junit:junit:4.10'
}
 
task createSourceDirs(description : 'Create empty source directories for all defined sourceSets') << {
    sourceSets*.allSource.srcDirs.flatten().each { File sourceDirectory ->
        if (!sourceDirectory.exists()) {
            println "Making \$sourceDirectory"
            sourceDirectory.mkdirs()
        }
    }
}
 
idea {
    project {
        jdkName = '1.6'
    }
}
EOF
 
    gradle createSourceDirs
 
    git init
    ls -a1 && find src    # list all created assets
}

It creates a build.gradle file ready to work with java and groovy projects, including IDEA integration (just execute gradle idea).

This gives you all of the tasks necessary to compile, jar, test, and distribute your code. For more information, check out the gradle docs on the java, groovy, and idea tasks.

It also creates all the necessary source directories for you and initializes a new git repository (with starting .gitignore file) for you to save your work.

You can easily tweak the build.gradle or .gitignore files to fit your needs. If you don’t use git, you can either delete those lines, or subsitute the equivalent lines for the source control tool you use. These are just a good starting place for me.

Here’s the sample output from the script above:

% mkdir testapp
% cd testapp
% newgradle                                                                   
Creating files for new gradle project
:createSourceDirs
Making /Users/tnaleid/Documents/workspace/testapp/src/main/resources
Making /Users/tnaleid/Documents/workspace/testapp/src/main/java
Making /Users/tnaleid/Documents/workspace/testapp/src/main/groovy
Making /Users/tnaleid/Documents/workspace/testapp/src/test/resources
Making /Users/tnaleid/Documents/workspace/testapp/src/test/java
Making /Users/tnaleid/Documents/workspace/testapp/src/test/groovy
 
BUILD SUCCESSFUL
 
Total time: 2.344 secs
Initialized empty Git repository in /Users/tnaleid/Documents/workspace/testapp/.git/
.
..
.git
.gitignore
.gradle
build.gradle
src
src
src/main
src/main/groovy
src/main/java
src/main/resources
src/test
src/test/groovy
src/test/java
src/test/resources

Then you’ve got all this gradle functionality ready to use:

% gradle tasks                                                           
:tasks
 
------------------------------------------------------------
All tasks runnable from root project
------------------------------------------------------------
 
Build tasks
-----------
assemble - Assembles all Jar, War, Zip, and Tar archives.
build - Assembles and tests this project.
buildDependents - Assembles and tests this project and all projects that depend on it.
buildNeeded - Assembles and tests this project and all projects it depends on.
classes - Assembles the main classes.
clean - Deletes the build directory.
jar - Assembles a jar archive containing the main classes.
testClasses - Assembles the test classes.
 
Documentation tasks
-------------------
groovydoc - Generates Groovydoc API documentation for the main source code.
javadoc - Generates Javadoc API documentation for the main source code.
 
Help tasks
----------
dependencies - Displays the dependencies of root project 'test'.
help - Displays a help message
projects - Displays the sub-projects of root project 'test'.
properties - Displays the properties of root project 'test'.
tasks - Displays the tasks runnable from root project 'test' (some of the displayed tasks may belong to subprojects).
 
IDE tasks
---------
cleanIdea - Cleans IDEA project files (IML, IPR)
idea - Generates IDEA project files (IML, IPR, IWS)
 
Verification tasks
------------------
check - Runs all checks.
test - Runs the unit tests.
 
Other tasks
-----------
cleanIdeaWorkspace
createSourceDirs - Create empty source directories for all defined sourceSets
 
Rules
-----
Pattern: build<ConfigurationName>: Assembles the artifacts of a configuration.
Pattern: upload<ConfigurationName>: Assembles and uploads the artifacts belonging to a configuration.
Pattern: clean<TaskName>: Cleans the output files of a task.
 
To see all tasks and more detail, run with --all.
 
BUILD SUCCESSFUL
 
Total time: 4.871 secs
2 Comments

Finding and Purging Big Files From Git History

2012/01/17

On a recent grails project, we’re using a git repo that was originally converted from a SVN repo with a ton of large binary objects in it (lots of jar files that really should come from an ivy/maven repo). The .git directory was over a gigabyte in size and this made it very cumbersome to clone and manipulate.

We decided to leverage git’s history rewriting capabilities to make a much smaller repository (and kept our previous repo as a backup just in case).

Here are a few questions/answers that I figured out how to answer with git and some shell commands:

What object SHA is associated with each file in the Repo?

Git has a unique SHA that it associates with each object (such as files which it calls blobs) throughout it’s history.

This helps us find that object and decide whether it’s worth deleting later on:

git rev-list --objects --all | sort -k 2 > allfileshas.txt

Take a look at the resulting allfileshas.txt file for the full list.

What Unique Files Exist Throughout The History of My Git Repo?

If you want to see the unique files throughout the history of your git repo (such as to grep for .jar files that you might have committed a while ago):

    git rev-list --objects --all | sort -k 2 | cut -f 2 -d\  | uniq

How Big Are The Files In My Repo?

We can find the big files in our repo by doing a git gc which makes git compact the archive and stores an index file that we can analyse.

Get the last object SHA for all committed files and sort them in biggest to smallest order:

git gc && git verify-pack -v .git/objects/pack/pack-*.idx | egrep "^\w+ blob\W+[0-9]+ [0-9]+ [0-9]+$" | sort -k 3 -n -r > bigobjects.txt

Take that result and iterate through each line of it to find the SHA, file size in bytes, and real file name (you also need the allfileshas.txt output file from above):

for SHA in `cut -f 1 -d\  < bigobjects.txt`; do
echo $(grep $SHA bigobjects.txt) $(grep $SHA allfileshas.txt) | awk '{print $1,$3,$7}' >> bigtosmall.txt
done;

(there’s probably a more efficient way to do this, but this was fast enough for my purposes with ~50k files in our repo)

Then, just take a look at the bigtosmall.txt file to see your biggest file culprits.

Purging a file or directory from history

Use filter-branch to remove the file/directory (replace MY-BIG-DIRECTORY-OR-FILE with the path that you’d like to delete relative to the root of the git repo:

git filter-branch --prune-empty --index-filter 'git rm -rf --cached --ignore-unmatch MY-BIG-DIRECTORY-OR-FILE' --tag-name-filter cat -- --all

Then clone the repo and make sure to not leave any hard links with:

git clone --no-hardlinks file:///Users/yourUser/your/full/repo/path repo-clone-name

You can use this command from the parent directory that contains your git repository and it’s clone to see how much space each of them take, and how much you’ve shrunk the repo in size:

du -s *(/)     # add the -h flag to see the output in human readable size formats, just like ls -lah vs ls -la

With these commands, I was able to reduce the file size of our repo with a few thousand commits down below the size of the checked out repository (more than an order of magnitude smaller). I only removed old binary files, we still have full history for all code files.

3 Comments

How to use kdiff3 as a 3-way merge tool with mercurial, git, and Tower.app

2012/01/12

There are a few very nice looking, mac-like diff tools for OSX (Kaleidoscope and Changes come to mind), but none for doing “real” merges. By this, I mean real, 3-way merges with all of the information you need in front of you.

There are no good-looking, “mac-like” merge tools, but if you swallow your pride there are a few different options for 3-way merges, including Araxis Merge ($$$!), DiffMerge, DeltaWalker, and FileMerge which comes free with XCode.

I’ve tried them all, and find them all confusing. They all tend use a 3-pane display to do the merging with your file in the left pane, the file you’re merging in the right pane, and the messy half-merged file in the middle.

That’s not enough information.

A 3-way merge actually has four important sources of information: Read the rest of this article »

1 Comment