linux-ext4 - Re: ext4 hang and per-memcg dirty throttling

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:   Wed, 12 Sep 2018 12:19:42 -0700
From:   Liu Bo <bo.liu@...ux.alibaba.com>
To:     Jan Kara <jack@...e.cz>
Cc:     linux-ext4@...r.kernel.org, fengguang.wu@...el.com, tj@...nel.org,
        cgroups@...r.kernel.org, gthelen@...gle.com, linux-mm@...ck.org,
        yang.shi@...ux.alibaba.com
Subject: Re: ext4 hang and per-memcg dirty throttling

On Wed, Sep 12, 2018 at 02:11:30PM +0200, Jan Kara wrote:
> Hi!
> 
> On Tue 11-09-18 17:10:55, Liu Bo wrote:
> > With ext4's data=ordered mode and the underlying blk throttle setting, we
> > can easily run to hang,
> > 
> > 1.
> > mount /dev/sdc /mnt -odata=ordered
> > 2.
> > mkdir /sys/fs/cgroup/unified/cg
> > 3.
> > echo "+io" > /sys/fs/cgroup/unified/cgroup.subtree_control
> > 4.
> > echo "`cat /sys/block/sdc/dev` wbps=$((1 << 20))" > /sys/fs/cgroup/unified/cg/io.max
> > 5.
> > echo $$ >  /sys/fs/cgroup/unified/cg/cgroup.procs
> > 6.
> > // background dirtier
> > xfs_io -f -c "pwrite 0 1G" $M/dummy &
> > 7.
> > echo $$ > /sys/fs/cgroup/unified/cgroup.procs
> > 8.
> > // issue synchronous IO
> > for i in `seq 1 100`;
> > do
> >     xfs_io -f -s -c "pwrite 0 4k" $M/foo > /dev/null
> > done
> > 
> > 
> > And the hang is like
> > 
> >       [jbd2-sdc]
> > jbd2_journal_commit_transaction                              
> >   journal_submit_data_buffers
> >     # file 'dummy' has been written by writeback kthread
> >   journal_finish_inode_data_buffers
> >     # wait on page's writeback
> 
> Yes, I guess you're speaking about the one Chris Mason mentioned [1].

Exactly.

> Essentially it's a priority inversion where jbd2 thread gets blocked behind
> writeback done on behalf of a heavily restricted process. It actually is
> not related to dirty throttling or anything like that. And the solution for
> this priority inversion is to use unwritten extents for writeback
> unconditionally as I wrote in that thread. The core of this is implemented
> and hidden behind dioread_nolock mount option but it needs some serious
> polishing work and testing...

Thank you so much for the details, so setting extent to unwritten and
then converting it in endio does work and keeps the data=ordered
semantic but I have to say the name, "dioread_nolock", is really
confusing...

thanks,
-liubo
> 
> [1] https://marc.info/?l=linux-fsdevel&m=151688776319077
> 
> 								Honza
> -- 
> Jan Kara <jack@...e.com>
> SUSE Labs, CR