Fast scanning with video and photogrammetry

Hi everyone,

Following a first experiment I wanted to share with the community to seek any interest/feedback in order to continue/enhance the development. Might spend more time on it if people are interested!

To give the context, MESH is working on picture gathered on internet to produce 3D models with photogrammetry. We need to deal with what we have, which is often not that great! Difference in exposition, light, contrast, time of day, quality, shadows etc..We are more focusing on processing with terrible databse rather than optimizing the scanning (Which would be way easier but we can’t!).

I had a try with a fast scanning processing, which consist on taking a video around an object without any intention to make a good video. Basically, I just start the recording on my camera (Olympus Stylus) while turning it toward the object. I didn’t looked at the screen nor checking the focus. I did a pretty bad job as a cameraman but it was the point!

Above are two frames from the resulting video where you can spot some obvious issues. I used Matlab to extract the frames in a basic fashion but it can be improve in many ways. If you take one frame every 20 for example, you might end up on the blurry one or with something blocking the view (like bellow). I am planning to add some selective process but one step at a time! The other way would be to extract in a fanatic way, like every frame, and sorting out by hand but I’m a lazy physicist..
Also depending on the speed and movement, it might be able to average the frame in order to increase the image quality, which I didn’t do in that case.

Once the frame extracted for both object, I did no further processing. We use enhancement while taking picture from internet and got sometime better result with it, but it was nice for once to work with pics that have the same format (:wink:). So directly into the photogrammetry softwares that were in that case VisualSFM and photoscan. Photoscan worked better, which is not always true for us (reader should not forget that most of our work is based on collected images!), VisualSFM still managed to get most of the object.

Here is the model of the stone:

And here is the boat:

Note that the boat was a bit more difficult since you have the canal, the tree and pillar around while the stone occupy the entire field of view. Also due to the canal, impossible to take a shoot in front of the boat as I didn’t fell like swimming during winter!

As you can see, both models could be improve with post-processing but as an experimental process the focus was on the video, if anyone want to join our adventure to refine the models we are open collaboration.

The stone model was quite straight forward, the first results were impressive.The object is a bit difficult as it contains many holes (which is the point of these stones by the way). We could think about extracting more frame around the holes, but as you can see on the first pics, the light was not the best to see inside features. The boat was more problematic, at first only half was modeled. I tried to select the “best pictures” (without anything in front) which resulted in a worst model. Turn out the best result was achieve with some bad ones (including even out of focus..).

I tried to use some markers from photoscan which worked but didn’t really improve the model. Also, these markers had to be manually corrected on each images which greatly increase the time to invest. Mask didn’t worked well neither but I did a quick job for that.

At the end, I was quite happy with the stone (minus the holes not modeled making holes into the model..tricky) and it appears that fastscan can be interesting to exploit even though it would require further work. If it is of interest to anybody, I would be happy to provide the code of frame selection once I spend time on it.

Nimrud assessment after liberation

Once journalists reached Nimrud, pictures started to be release online of what has been left. Expectation was low: following the video featuring a huge explosion, we could imagine nothing was left. The pictures and videos do show a chaotic place with rubbles, bricks and other materials. However, some pictures are encouraging, others worrying.

Here is a picture before and after the entrance. The left gate is often portrayed to represent Nimrud as you might have seen in several newspapers and is now gone.


Entrance before (up) and after (bottom)

The right gate looks untouched by the blast but a close look shows that the Lamassus and the relief (expect one) have been extracted and stacked on the ground.


The stack that you can see in the first picture and in several videos reveals the rest of this right gate. For now there are no signs of remains of the left gate from the pictures we got but they might be beneath the rest.



And here is the only untouched relief from the right gate. It is particularly interesting because it does not show the severe damage caused by intentional blasts, unlike the Lamassus on the left. The important fracture that cuts the relief in half was already there, only the restoration material went away, certainly due to shocks.


But there is an encouraging aspect as many blocks and reliefs seem to have been laid down to the ground without being completly destroy. Some did not even move after the video was taken as we can still see the stack in front of the entrance. It raises the question: did ISIS have the time or the interest to do something else with it? Such as selling stuff on the black market? The damage must be assessed with a consideration of the previous poor states of the relief and Lamassus. The pictures before the war already show obvious fractures. Visitors to the British museum can also see such damages. Some of the fracture that can be seen (in the pictures above and below) already exist prior the destruction. Others (at the bottom of the picture below) are new. The good news is that it can be fixed, if one can find the other pieces in a “good enough” state.


On the other hand, many pictures featuring rubbles of the place do not show any remains of the relief that would have been blasted during the explosion. To be sure of that, archaeologists will have to sort out all pieces. But if the pieces of relief are missing, it probably indicates an extraction for antiquities trafficking. It is also possible that they ended up in Mosul and ISIS did not have time to sell it on the black market.

The pictures were acquired from Max Delany, AFP correspond, also from our contact in Iraq, social media. Thanks again to Suzanne Bott for providing the picture prior to damage.


Printing models

Photogrammetry produces a nice visual 3D model which can be eventually printed since 3D printers become more and more accessible. However, there is a strong difference between a model for digital display and for printing. The first one has a texture than can enhance the impression of 3D while the surface might not have any shape, which can be seen with the dense point.


Figure 1: Difference before and after processing of the Nimrud entrance

After photogrammetry it is usual to rework the model in order to make it printable. In some case textures or known features on the object can help this task but it requires 3D modeling skills.

During the 2nd SEAHA conference (a while now), we attached a printed model to our poster. Our goal was to offer another way to evaluate the impact of our processing than the graph and the 2D visuals. We tried to print both enhanced and non-enhanced and demonstrates how our methods reduce noise and highlight real shape of the object.


Figure 2: 3D model original (left) and enhanced after image processing (right)

But the non-enhanced model had way too much noise and took too much time to be printed before the event. That is why we only got the enhanced model and we are showing here a comparison of the previews before printing.

Drones, a new adventure

Like many, our work on 3D modelling led us to a strong interest in drones’ technologies. These devices get cheaper and cheaper over time while their manipulation get easier for non-expert. But one has to keep in mind that reasonable use of drones requires knowledge and skills. Instead of jumping directly in this field, we decided to collaborate with a group already involved in use of drones. DroneArcheology [1] is based in the US and got a strong expertise regarding South American heritage, in particular with their expedition deep in the jungle to find old tomb with drone support.


Canoa after the earthquake

However, we didn’t start our collaboration on such ancient site, Indiana Jones 2.0 is for later. Now we are looking at using drone to support reconstruction plan after the earthquake in Ecuador. This is linked with a bigger project looking at drone application for humanitarian purposes [2]. DroneArcheology produce a lot of data and we are helping to process. We are ourselfes in a very stage of learning such application as well and we keep our eyes open for innovative process.

One thing we are really looking forward is the application of underwater drones. A company called OpenROV [3] made an open source version that got quickly popular. DroneArcheology is using these machine since their early stage which make an amazing opportunity to be involve the improvement for heritage application.




Yahya Shrine

The ongoing war in Iraq results in disastrous condition of the citizens and the heritage sites as it has been mentioned numerous time though media and official channels. Everyone heard about the museum in Mosul and the blast of the palace of Ashurnasirpal II in Nimrud. But there are countless other smaller sites that are gone. Monument of Mosul in Danger [1], a project based in Prague survey the list of lost monument in the ancient city which reach 38 entries. One of them is quite interesting for its original roof and more importantly its history.

We proposed our help to see if it was possible to produce a model for documentation purpose. They liked the idea and share information with us, but it seems that we were quite optimistic on that one!


Figure 1: Two different pictures of the shrine showing different quality and limitations

The few pictures we received are quite different. One main issue is the modification occurring on the building overtime due to repair. This rise the problematic to know which state of the monument we should digitally preserved, if not all known.

With floor maps and few pictures, we were able to make a model of the outside. The outside structure is quite simple and the building can be easily recognised with its particular roof, the two massive buttres that protected to fall into the river and the main entrance. However, once the model made we have to apply the textures. The modern modifications hide many details due to time damage that reflect the history of the monument and it could be interesting to have both versions. It is still hard to get details pictures and we are currently working on it.


Figure 2: A quick try from map floor..a lot of work ahead!

Meanwhile, we can notice a particular link with another important site of Mosul. In the Islamic gallery of the museum, there is/was a chest from the Yahya Shrine. This gallery has not been display in the last footage about the museum and we still don’t know what happened. Let see what the future will tell us.



Process optimisation

As a first article, it is hard to avoid mentioning the starting point of Mesh related to lost heritage. We were collecting data of lost heritage to produce 3D model using photogrammetry, a well-known technique applied daily in heritage as you can find many example on Sketchfab [1] but usually rely on controlled acquisition of the pictures.

In the case of lost heritage, pictures’ sources vary and we find different quality and setup. They are also taken at different time, with various weather condition which impact the light exposition of the subject to model. Additionally, some pictures are black and white or even only drawing that prevent the direct use of photogrammetry.

We’ve decided to look at various ways to enhance the process, share the experience online, exchange with other researchers and improve the work. We believe that communication is a key for effective development and will have a benefit impact on the applicability of the result.

One of the first contribution of Mesh was a simple post on the heritage research network [2] that highlights some of the issues which can be find now on our website such as lost heritage and lootings. From that we went forward, increase our list of contacts, projects and also our first poster (figure 1) at the second conference in Science and Engineering for art, heritage and archaeology.


Figure 1: Poster at the 2nd SEAHA conference

It is important to notice that our work is entirely volunteer. Therefore, we are not pushing the producing of outputs that will come in time. We rather prefer a continuous work that make sense and can be follow by anyone while encouraging participation and exchange.