Re: [cf-dev] Purge files on NFS or S3?

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: [cf-dev] Purge files on NFS or S3?

James Bayer
john, i think the resource files may grow forever right now without intervention.

i'm pretty confident that when apps are deleted that their droplets are deleted with them and that proper garbage collection occurs with that.

i'm unaware of any NFS file system to s3 blob migration. you would need to update the CC_DB references too i'm pretty sure. i'm interested if you find out more.

On Tue, May 5, 2015 at 1:14 PM, John Wong <[hidden email]> wrote:
Hi

I just looked at our disk usage on NFS server. We have used like 200G so far, and I wonder if there's a systematic way to purge files we don't need (or how do I know I don't need them)?

Similarly, if I were to replace NFS server with S3 instead, does the existing process (if any) work with S3?

Thanks.

_______________________________________________
cf-dev mailing list
[hidden email]
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev




--
Thank you,

James Bayer

_______________________________________________
cf-dev mailing list
[hidden email]
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
Reply | Threaded
Open this post in threaded view
|

Re: [cf-dev] Purge files on NFS or S3?

Chad Woolley
Not sure if this is what you need, but you can manually sync + delete files from a local filesystem (including NFS mount) to/from S3:

http://s3tools.org/s3cmd-sync

... with `—delete-removed` option

-- Chad

On Sat, May 9, 2015 at 12:19 AM, James Bayer <[hidden email]> wrote:

>
> john, i think the resource files may grow forever right now without intervention.
>
> i'm pretty confident that when apps are deleted that their droplets are deleted with them and that proper garbage collection occurs with that.
>
> i'm unaware of any NFS file system to s3 blob migration. you would need to update the CC_DB references too i'm pretty sure. i'm interested if you find out more.
>
> On Tue, May 5, 2015 at 1:14 PM, John Wong <[hidden email]> wrote:
>>
>> Hi
>>
>> I just looked at our disk usage on NFS server. We have used like 200G so far, and I wonder if there's a systematic way to purge files we don't need (or how do I know I don't need them)?
>>
>> Similarly, if I were to replace NFS server with S3 instead, does the existing process (if any) work with S3?
>>
>> Thanks.
>>
>> _______________________________________________
>> cf-dev mailing list
>> [hidden email]
>> https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
>>
>
>
>
> --
> Thank you,
>
> James Bayer
>
> _______________________________________________
> cf-dev mailing list
> [hidden email]
> https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
>

_______________________________________________
cf-dev mailing list
[hidden email]
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
Reply | Threaded
Open this post in threaded view
|

Re: [cf-dev] Purge files on NFS or S3?

John Wong
Hi all

Thanks. No I was just curious if there was a way to identify what to remove in the blobstore because I was surprised the size of my blobstore at this point. I will check what's in there (maybe James is right it is mostly resource files). I am currently using NFS. I can build a CF with S3 as my blobstore.

John


On Mon, May 11, 2015 at 11:36 AM, Chad Woolley <[hidden email]> wrote:
Not sure if this is what you need, but you can manually sync + delete files from a local filesystem (including NFS mount) to/from S3:

http://s3tools.org/s3cmd-sync

... with `—delete-removed` option

-- Chad


On Sat, May 9, 2015 at 12:19 AM, James Bayer <[hidden email]> wrote:

>
> john, i think the resource files may grow forever right now without intervention.
>
> i'm pretty confident that when apps are deleted that their droplets are deleted with them and that proper garbage collection occurs with that.
>
> i'm unaware of any NFS file system to s3 blob migration. you would need to update the CC_DB references too i'm pretty sure. i'm interested if you find out more.
>
> On Tue, May 5, 2015 at 1:14 PM, John Wong <[hidden email]> wrote:
>>
>> Hi
>>
>> I just looked at our disk usage on NFS server. We have used like 200G so far, and I wonder if there's a systematic way to purge files we don't need (or how do I know I don't need them)?
>>
>> Similarly, if I were to replace NFS server with S3 instead, does the existing process (if any) work with S3?
>>
>> Thanks.
>>
>> _______________________________________________
>> cf-dev mailing list
>> [hidden email]
>> https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
>>
>
>
>
> --
> Thank you,
>
> James Bayer
>
> _______________________________________________
> cf-dev mailing list
> [hidden email]
> https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
>

_______________________________________________
cf-dev mailing list
[hidden email]
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev



_______________________________________________
cf-dev mailing list
[hidden email]
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
Reply | Threaded
Open this post in threaded view
|

Re: [cf-dev] Purge files on NFS or S3?

Dieu Cao
An option could be to just delete all the resource files on the blobstore. The effect would be that for binaries that would have been matched, they would be uploaded again on the first new push including those binaries.

On Monday, May 11, 2015, John Wong <[hidden email]> wrote:
Hi all

Thanks. No I was just curious if there was a way to identify what to remove in the blobstore because I was surprised the size of my blobstore at this point. I will check what's in there (maybe James is right it is mostly resource files). I am currently using NFS. I can build a CF with S3 as my blobstore.

John


On Mon, May 11, 2015 at 11:36 AM, Chad Woolley <<a href="javascript:_e(%7B%7D,&#39;cvml&#39;,&#39;thewoolleyman@gmail.com&#39;);" target="_blank">thewoolleyman@...> wrote:
Not sure if this is what you need, but you can manually sync + delete files from a local filesystem (including NFS mount) to/from S3:

http://s3tools.org/s3cmd-sync

... with `—delete-removed` option

-- Chad


On Sat, May 9, 2015 at 12:19 AM, James Bayer <<a href="javascript:_e(%7B%7D,&#39;cvml&#39;,&#39;jbayer@pivotal.io&#39;);" target="_blank">jbayer@...> wrote:

>
> john, i think the resource files may grow forever right now without intervention.
>
> i'm pretty confident that when apps are deleted that their droplets are deleted with them and that proper garbage collection occurs with that.
>
> i'm unaware of any NFS file system to s3 blob migration. you would need to update the CC_DB references too i'm pretty sure. i'm interested if you find out more.
>
> On Tue, May 5, 2015 at 1:14 PM, John Wong <<a href="javascript:_e(%7B%7D,&#39;cvml&#39;,&#39;gokoproject@gmail.com&#39;);" target="_blank">gokoproject@...> wrote:
>>
>> Hi
>>
>> I just looked at our disk usage on NFS server. We have used like 200G so far, and I wonder if there's a systematic way to purge files we don't need (or how do I know I don't need them)?
>>
>> Similarly, if I were to replace NFS server with S3 instead, does the existing process (if any) work with S3?
>>
>> Thanks.
>>
>> _______________________________________________
>> cf-dev mailing list
>> <a href="javascript:_e(%7B%7D,&#39;cvml&#39;,&#39;cf-dev@lists.cloudfoundry.org&#39;);" target="_blank">cf-dev@...
>> https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
>>
>
>
>
> --
> Thank you,
>
> James Bayer
>
> _______________________________________________
> cf-dev mailing list
> <a href="javascript:_e(%7B%7D,&#39;cvml&#39;,&#39;cf-dev@lists.cloudfoundry.org&#39;);" target="_blank">cf-dev@...
> https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
>

_______________________________________________
cf-dev mailing list
<a href="javascript:_e(%7B%7D,&#39;cvml&#39;,&#39;cf-dev@lists.cloudfoundry.org&#39;);" target="_blank">cf-dev@...
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev



_______________________________________________
cf-dev mailing list
[hidden email]
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
Reply | Threaded
Open this post in threaded view
|

Re: [cf-dev] Purge files on NFS or S3?

Jon Price
In reply to this post by James Bayer
Make sure you only delete the resource files, not everything...

Jon Price
Intel Corp.

On May 11, 2015 10:05 PM, Dieu Cao <[hidden email]> wrote:
An option could be to just delete all the resource files on the blobstore. The effect would be that for binaries that would have been matched, they would be uploaded again on the first new push including those binaries.

On Monday, May 11, 2015, John Wong <[hidden email]<mailto:[hidden email]>> wrote:
Hi all

Thanks. No I was just curious if there was a way to identify what to remove in the blobstore because I was surprised the size of my blobstore at this point. I will check what's in there (maybe James is right it is mostly resource files). I am currently using NFS. I can build a CF with S3 as my blobstore.

John


On Mon, May 11, 2015 at 11:36 AM, Chad Woolley <[hidden email]> wrote:
Not sure if this is what you need, but you can manually sync + delete files from a local filesystem (including NFS mount) to/from S3:

http://s3tools.org/s3cmd-sync

... with `—delete-removed` option

-- Chad


On Sat, May 9, 2015 at 12:19 AM, James Bayer <[hidden email]> wrote:

>
> john, i think the resource files may grow forever right now without intervention.
>
> i'm pretty confident that when apps are deleted that their droplets are deleted with them and that proper garbage collection occurs with that.
>
> i'm unaware of any NFS file system to s3 blob migration. you would need to update the CC_DB references too i'm pretty sure. i'm interested if you find out more.
>
> On Tue, May 5, 2015 at 1:14 PM, John Wong <[hidden email]> wrote:
>>
>> Hi
>>
>> I just looked at our disk usage on NFS server. We have used like 200G so far, and I wonder if there's a systematic way to purge files we don't need (or how do I know I don't need them)?
>>
>> Similarly, if I were to replace NFS server with S3 instead, does the existing process (if any) work with S3?
>>
>> Thanks.
>>
>> _______________________________________________
>> cf-dev mailing list
>> [hidden email]
>> https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
>>
>
>
>
> --
> Thank you,
>
> James Bayer
>
> _______________________________________________
> cf-dev mailing list
> [hidden email]
> https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
>

_______________________________________________
cf-dev mailing list
[hidden email]
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


_______________________________________________
cf-dev mailing list
[hidden email]
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev